SCLS Pilot Test 2015 Responses to Passback

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: March 13, 2015

TO: Shelly Martinez, OMB

THROUGH: Kashka Kubzdela, NCES

FROM: Isaiah O’Rear, NCES

SUBJECT: School Climate Surveys (SCLS) Pilot Test 2015 Update Responses to OMB Passback (OMB # 1850-0803 v.129)

Provided below are responses to OMB passback for the School Climate Surveys (SCLS) Pilot Test 2015 Update submission (OMB # 1850-0803 v.129).

1. What age group is getting the following questions and also please confirm that you’ve run these questions by Mike Planty and team at BJS since they are planning a campus sexual assault survey and may have better wording for you: “Two items below have been added to the student questionnaire per request from the Office of Safe and Healthy Students (OSHS). As a result, volume I, attachment 1 (questionnaires), and attachment 2 (justification tables) have been updated:

a. Sengrel153: At this school, there is a teacher or some other adult who students can go to if they need help because of sexual assault or dating violence.

b. Ssafpsaf152: Students at this school have been victims of sexual assault or dating violence.”

NCES: The sexual assault questions are asked of students in the high school survey, Grades 9-12. Mike Planty has reviewed the items regarding their consistency with a planned campus sexual assault survey. In response to Mike Planty’s comments, we are removing question Ssafpsaf152, because it does not adequately measure prevalence of sexual assault or measure school climate related to sexual assault. He also voiced a concern about how well Sengrel153 fits a 4-option Likert-scale response format. Because several items within the survey use the same format and performed well in the cognitive labs, we will keep the item unchanged for the pilot test. Also, the existing "Postsecondary Crime Survey," only asks about incidents of sexual assault at the institution level and has no overlap with SCLS questions.

2. When this project was last at OMB, I believe we asked for much more detailed information about how the survey would work with regard to ability to conduct nonresponse follow up and bias analysis. If the LEA or school is “in charge” how is NCES advising them so that this doesn’t become a good questionnaire but inadequate survey administration?

NCES: On the topic of nonresponse bias the platform has the functionality to monitor submission rates as the survey is being administered. Currently in the Administration and Technical Guide, we include text on “monitoring submission rates,” “non-response intervention options,” and some strategies for reaching out to parents given that, based on our experience and interviews with districts conducting similar surveys, parents are usually the most difficult to reach group (Attachment 4, pages TA-45 and TA-46). In the guide, we also emphasize that high response rates are very important for obtaining valid and unbiased data (page TA-43).

We have also provided the capacity for the administrator to conduct a nonresponse bias analysis (NRBA). The surveys have items to collect data on sex and race/ethnicity of the respondent, and for students, grade is also collected. We have included the univariate distributions of sex, race/ethnicity, and grade of enrollment along with two way cross-tabulations of these variables in the report (they will also be available in the downloadable spreadsheets of results). In addition, we have included instructions on how to use this information in conjunction with data the administrator has on the population being surveyed to compare the distributions for respondents to those of the population.

To facilitate the NRBA, a spreadsheet (Attachment 5) is provided for use in comparing the distribution of respondent characteristics to population characteristics of the school, school district, or state. The spreadsheet includes the necessary formulas to generate measures of bias, with instructions on how to load the two sets of population distributions (i.e., respondents and full population), and how to interpret the results. If the administrator is at the district level or the state level, they will have additional information on the school and/or district characteristics in their study population (e.g., the information routinely reported to CCD at the school and district levels) and they will have the ability to expand their NRBA. We have included guidance in the documentation on how to conduct these analyses (see Attachment 4).

We have added the following suggestions and discussion about nonresponse bias to the guide (Attachment 4, Page TA-46), and we will provide pilot schools with the frequency distributions of the demographic variables in the scale report we prepare for them after receiving the pilot data:

Nonresponse Bias:

• When the response rate is below 80 percent, a nonresponse bias analysis is recommended to determine whether or not the respondents to your study are representative of the population in your school, district, or state and to assess the potential magnitude of nonresponse bias. The analysis will help evaluate whether the data or reports based on the data are biased by the missing respondents.

• SCLS administrators can use the frequency distributions of the demographic variables (grade [student only], race/ethnicity, and sex) that are included in the reports and compare them to another data source that has the frequencies of these demographic variables for the total population. The non-response bias worksheet with embedded formulas is included in the SCLS package and can be used to carry out basic analysis. A worksheet for student survey will look like this [spreadsheet image is included in user’s guide]. The “Percent of respondents” column indicates the characteristics of the respondents to the survey. The “Percent of students” is extant information from the administrative data of the school or school system. The SCLS administrator should enter the demographic characteristics included at the end of item frequency report into the second column, labeled “Percent of respondents” and comparable data from the administrator’s records should be entered into the third column, labeled “Percent of students” in the example (this should also be used for the other categories of respondents). The “estimated bias” column indicates, in percentage point terms, differences between respondent and overall student population differences. The “relative bias column” indicates how large the bias is relative to the estimates from the “percent of respondents” column.

• In those cases in which the administrator has elected to preserve the identity of the students for the purpose of linking to data from their record systems, student survey responses can also be linked to external data sources to conduct more detailed bias analysis using additional student data to measure bias within the responding population as compared to the full population. Additionally, if the data are being collected at the district or state level, the administrator can add additional school and district data for additional analyses. If these additional data are used, the administrator would add the additional variables to the first column labeled [Student] characteristic, and drag the formulas in the fourth and fifth columns to the row corresponding to the end of the list of characteristics.

• The bias is computed by subtracting each value in the “Percent of [students]” column from the comparable value in the “Percent of respondents” column. The relative bias is the bias estimate for each row divided by “Percent of respondents”. For any group of respondents, if the estimated bias is larger than 1 percentage point (greater than 1.0 or less than -1.0), the survey data should be used with caution. Administrators should also be cautious if the relative bias is larger than 0.3 or less than -0.3. We can provide assistance to schools and districts in the pilot with their nonresponse bias analysis if needed.

3. To clarify, the district can access the individual results of individual participants in all of the surveys? And for students, use those identifiers to easily link individual survey responses to other student data? Is the concept of statistical use only embedded anywhere in this model? How?

NCES: SCLS administrators can export data and view the individual results of participants in all of the surveys. For students, administrators can use usernames as identifiers to link individual survey responses to other student data if the administrators match the usernames with students’ IDs or names outside of the SCLS platform before the survey administration.

Because SCLS does not ask PPRA prohibited questions, none of the data collected are about specific students' activities; instead the students are asked about students in their school. As a result the data are not suitable to be used to take action relative to individual students; however states, districts, and/or schools that administer the school climate survey, may use the resulting data for programmatic purposes. In such cases, the data would not be used for strictly statistical purposes. The survey platform for general distribution will be made available to states, districts, and schools through a website administered by the Office of Safe and Healthy Students, labeled as an OSHS product. It will not be available from an NCES website. All references to ‘NCES’ in the tool have been replaced with references to the ‘U.S. Department of Education’. Please note that in the case of the pilot and the national benchmarking surveys, reference is made to NCES collecting the data on behalf of ED or OSHS. Please let us know if you would prefer to have NCES removed in those instances.

However, because during the administration of the pilot test and the national benchmarking study NCES will receive and analyze the data, the confidentiality pledges provided to all respondents have been amended to include both the district or school’s use of the data and the use of the data by NCES for statistical purposes (please see the response to question 4).

4. Where did the confidentiality language for the local administrator come from? Has it been through legal review?

NCES: There are several locations of confidentiality language within the survey. The first two listed below are for use by the test administrator and proctors who work with students taking the survey. They are akin to the NCES affidavits of nondisclosure. The remaining entries are more like the traditional confidentiality pledges that are shared with respondents.

1. SURVEY ADMINISTATORS: Before using the platform, survey administrators are required to read and attest to the following confidentiality pledge when they set up SCLS and each time they export survey results. This confidentiality pledge serves the same function as an affidavit of nondisclosure (shown as Appendix C of the guide in the OMB# 1850 0803 v.119 package). This pledge is not viewed by respondents:

“I hereby certify that I have carefully read and will cooperate fully with the SCLS procedures on confidentiality. I will keep completely confidential all information arising from surveys concerning individual respondents to which I may gain access. I will not discuss, disclose, disseminate, or provide access to survey data and identifiers. I will devote my best efforts to ensure that there is compliance with the required procedures by personnel whom I supervise. I give my personal pledge that I shall abide by this assurance of confidentiality.

[checkbox] I have read and agree to the Pledge of Confidentiality.”

2. SURVEY PROCTORS: Included in the user’s guide is a sample page for administrators to have the survey proctors sign. The following is the language of the sample confidentiality pledge for survey proctors (located in Appendix E):

“I hereby certify that I have carefully read and will cooperate fully with the SCLS procedures on confidentiality. I will keep completely confidential all information arising from surveys concerning individual respondents to which I may gain access. I will not discuss, disclose, disseminate, or provide access to survey data and identifiers. I give my personal pledge that I shall abide by this assurance of confidentiality.

My signature below indicates that I have read and agree to the Pledge of Confidentiality.

Signature: Date:

Printed Name”

3. STUDENT RESPONDENTS: For the pilot and benchmarking studies the following confidentiality language will be presented to student respondents within the survey platform informing students of the potential uses of their data, including the role of NCES in analyzing the data. Survey administrators will not be able to remove or edit this language:

“The experiences of students are critical to understanding school climate, and the best way to understand those experiences is to ask students themselves. Apart from improving your school climate, the information you provide will be used by the National Center for Education Statistics (NCES) to improve questionnaires to be answered by students like yourself in the future. Data provided to NCES may be used only for statistical purposes and may not be disclosed or used in identifiable form for any other purpose, except as required by law (Education Sciences Reform Act (ESRA) of 2002, 20 U.S.C., § 9573).”

Because the survey administrator has the option of preserving a link to individual students among the survey respondents, the survey administrator will be required to select between two options for the wording of the pledge—Version A for administrators who are not preserving a link to student records and Version B for administrators who are preserving the link for future analysis. Apart from the selection of Version A or B, survey administrators will not be able to remove or edit this language

Version A of Confidentiality pledge offered to students when links to student records are not preserved:

“The data you provide may also be used by your school and district to better understand the current climate in your school. The only people who will see your answers to individual questions are authorized personnel at your school and district (20 U.S.C. § 1232g; 34 CFR Part 99). Your answers will be combined with the answers of other students at your school and district and used to create reports about the climate of your school. These reports will not identify any person or their responses.”

Version B of Confidentiality pledge offered to students when links to student records are preserved:

“The data you provide may also be used by your school and district to better understand the current climate in your school. The only people who will see your answers to individual questions are authorized personnel at your school and district (20 U.S.C. § 1232g; 34 CFR Part 99). Your answers will be combined with the answers of other students at your school and district and used to create reports about the climate of your school. Although these reports will not identify any person or their responses, your data may be combined with other data about you to help your school to improve the climate of your school.”

Administrators will have the ability to add customized text to the end of this confidentiality language where they can add references to relevant state and local confidentiality laws and/or policies.

The user’s guide will also include two sample parent consent forms for schools in order to obtain the consent of parents for administering the student respondent survey (located in Appendix G):

Appendix G.1

Sample Opt-Out Parental Consent Form I (Opt in NOT Required)

[NOTE: While the italicized text may be modified to suit the needs of each school or district, the non-italicized text should be kept consistent across all pilot study sites.]

Dear parent/guardian:

<SCHOOL NAME> is participating in a pilot test of a voluntary survey about school climate. The survey will be administered to students during regular school hours during the week of <WEEK OF ADMINISTRATION>. It will ask students about their perceptions surrounding topics such as student engagement, school environment, and school safety.

To analyze and refine the questionnaires, the National Center for Education Statistics (NCES) will receive individual-level responses from participating schools and districts but without the names or other direct personal identifiers of the respondents. All information received by NCES that in any way relates to or describes identifiable characteristics of individuals is protected from disclosure by federal statute; it may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act (ESRA) of 2002, 20 U.S.C., § 9573). If you want additional information about NCES’s use of your child’s data please contact <NCES CONTACT INFO>.

The data your child provides may also be used by their school and district to better understand the current climate in their school. The only people who will see your child's answers to individual questions are authorized personnel at their school and district (20 U.S.C. § 1232g; 34 CFR Part 99). Your child's answers will be combined with the answers of other students at their school and district and used to create records about the climate of their school. These reports will not identify any person or their responses.

<USE IF LINKING STUDENT RECORDS>:

Although these reports will not identify any person or their responses, your child's data may be combined with other data about your child to help their school to improve the climate of their school.

If you do not want your child to participate in this survey, please complete, sign, and postmark this letter using the enclosed postage paid envelope by <DEADLINE DATE>. If you sign and postmark this letter by <DEADLINE DATE>, this means your child will not participate in the survey and will be asked to report to a designated place in the school (for example, the library) while the survey is administered.

If you have any questions about this study or about your child’s participation or would like to see a copy of the student survey, please contact <ADMINISTRATOR NAME> at <ADMINISTRATOR NUMBER>. If you have any questions about your child’s rights as a participant in this study, please contact <SCHOOL CONTACT INFO>.

I understand that by completing and signing the form below and returning this letter, my child will not be allowed to take the School Climate Survey.

Parent/Guardian Signature: Date:

Student Signature: Date:

Student Printed Name: Date of birth:

If you agree to allow your child to take the survey, you do not have to sign or send back anything.

Sincerely,

<ADMINISTRATOR NAME>

Appendix G.2

Sample Opt-In Parental Consent Form II (Opt in Required)

[NOTE: While the italicized text may be modified to suit the needs of each school or district, the non-italicized text should be kept consistent across all pilot study sites.]

Dear parent/guardian:

<SCHOOL NAME> is participating in a pilot test of a voluntary survey about school climate. The survey will be administered to students during regular school hours during the week of <WEEK OF ADMINISTRATION>. It will ask students about their perceptions surrounding topics such as student engagement, school environment, and school safety.

To analyze and refine the questionnaires, the National Center for Education Statistics (NCES) will receive individual-level responses from participating schools and districts but without the names or other direct personal identifiers of the respondents. All information received by NCES that in any way relates to or describes identifiable characteristics of individuals is protected from disclosure by federal statute; it may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act (ESRA) of 2002, 20 U.S.C., § 9573). If you want additional information about NCES’s use of your child’s data please contact <NCES CONTACT INFO>.

The data your child provides may also be used by their school and district to better understand the current climate in their school. The only people who will see your child's answers to individual questions are authorized personnel at their school and district (20 U.S.C. § 1232g; 34 CFR Part 99). Your child's answers will be combined with the answers of other students at their school and district and used to create records about the climate of their school. These reports will not identify any person or their responses.

<USE IF LINKING STUDENT RECORDS>:

Although these reports will not identify any person or their responses, your child's data may be combined with other data about your child to help their school to improve the climate of their school.

If you agree to allow your child to participate in this survey, please complete, sign, and postmark this letter using the enclosed postage paid envelope by <DEADLINE DATE>. If you sign and postmark this letter by <DEADLINE DATE>, this means your child will participate in the survey and will be asked to report to a designated place in the school (for example, the computer lab) where the survey is administered.

If you have any questions about this study or about your child’s participation or would like to see a copy of the student survey, please contact <ADMINISTRATOR NAME> at <ADMINISTRATOR NUMBER>. If you have any questions about your child’s rights as a participant in this study, please contact <SCHOOL CONTACT INFO>.

I understand that by completing and signing the form below and returning this letter, my child will be allowed to take the School Climate Survey.

Parent/Guardian Signature: Date:

Student Signature: Date:

Student Printed Name: Date of birth:

If you do not want your child to take the survey, you do not have to sign or send back anything.

Sincerely,

<ADMINISTRATOR NAME>

4. ADULT RESPONDENTS. For the pilot and benchmarking studies the following confidentiality language will be presented to teachers, other staff, and parent respondents.

For Staff: “The experiences of students, staff and parents are critical to understanding school climate, and the best way to understand those experiences is to ask all members of the school community. Apart from improving your school climate, the information you provide will be used by the National Center for Education Statistics (NCES) to improve questionnaires to be answered by people like yourself in the future. NCES will receive individual-level responses from participating schools and districts without the names or other direct personal identifiers of the respondents. Data provided to NCES may be used only for statistical purposes and may not be disclosed or used in identifiable form for any other purpose, except as required by law (Education Sciences Reform Act (ESRA) of 2002, 20 U.S.C., § 9573).

The data you provide may also be used by your school and district to better understand the current climate in your school. The only people who will see your answers to individual questions are authorized personnel at your school and district. Your answers will be combined with the answers of other respondents in your school and district and used to create reports about the climate of your school. These reports will not identify any person or their responses.”

For Parents: “The experiences of students, staff and parents are critical to understanding school climate, and the best way to understand those experiences is to ask all members of the school community. Apart from improving your child’s school climate, the information you provide will be used by the National Center for Education Statistics (NCES) to improve questionnaires to be answered by people like yourself in the future. NCES will receive individual-level responses from participating schools and districts without the names or other direct personal identifiers of the respondents. Data provided to NCES may be used only for statistical purposes and may not be disclosed or used in identifiable form for any other purpose, except as required by law (Education Sciences Reform Act (ESRA) of 2002, 20 U.S.C., § 9573).

The data you provide may also be used by your child’s school and district to better understand the current climate in the school. The only people who will see your answers to individual questions are authorized personnel at your child’s school and district. Your answers will be combined with the answers of other respondents in your child’s school and district and used to create reports about the climate of the school. These reports will not identify any person or their responses.”

Administrators will have the ability to add customized text to this confidentiality language where they can add references to relevant state and local confidentiality laws.

5. Why do you think it is appropriate to provide a $500 incentive (versus collecting the items at the end of the survey)? This seems quite generous given the burden and the fact that the school gets results to keep.

NCES: OMB approved the $500 incentive in December in the SCLS Pilot 2015 request (OMB# 1850-0803 v.119), to which this submission provides only minor updates. The approved incentive has been communicated to potential respondents during the currently ongoing recruitment process (per the December clearance). The fact that the survey will be administered to all students, staff, and parents poses a significantly greater burden than a typical subsample survey. Although schools can also benefit from the results, we are asking them to pilot test this survey with all respondents on a timeframe necessitated by our developmental schedule. In addition, because SCLS is online-based with no paper and pencil options, we would especially like to attract schools with less technical capacity to participate in the pilot test. With the extra computer, they will have the technology to make the survey available to parents and guardians during back to school nights, as suggested in the guide. Schools will be able to use the equipment for this pilot test, as well as for future administrations of SCLS or other technical needs.