Justification

National Center for Education Statistics

Volume I

Supporting Statement

National Center for Education Statistics (NCES) Confidentiality Pledges Mail Experiment

OMB# 1850-0803 v.220

November 2017

Submittal –Related Information

This material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (e.g., focus groups, cognitive interviews, usability tests, experiments, etc.) to develop and test study materials and methodologies so as to improve future data quality, utility, and study participant experience.

1. Background

The passage of the Cybersecurity Enhancement Act of 2015 required the installation of the Department of Homeland Security (DHS)’s Einstein cybersecurity protection system on all Federal civilian information technology systems by mid-December 2016. As a result, the confidentiality pledge provided to respondents at the start of NCES surveys was updated to reflect the new legislation.

This request is to conduct a controlled experiment under simulated survey conditions that builds on the results of cognitive testing conducted in 2016 (OMB# 1850-0803 v.162). The prior research was conducted to evaluate revised confidentiality pledge wording with regard to respondents’ comprehension and reactions, and explore wording options that may minimize negative impacts. Participants in the cognitive testing expressed concerns about who would be involved in monitoring online survey collections, and also tended to have positive reactions to language describing penalties for Federal employees for disclosing personally identifiable information. The goal of the experiment proposed here is to test the effects on response of specifying that DHS will be involved in monitoring data collection systems, and of statements specifying penalties for government employees who disclose respondents’ information. This survey specifically targets institutional respondents, namely school administrators. The Center for Survey Measurement (CSM) at the Census Bureau will coordinate the collection of the experiment survey by operations staff at the Census Bureau’s National Processing Center. The specific confidentiality pledges and penalties statements that will be tested are provided in Attachment 1 in this document.

2. Methodology

We will conduct a survey based on the 2017-18 National Teachers and Principals Survey (NTPS). NTPS is a voluntary, multi-phase survey involving teachers and principals, but this experiment will be conducted only using the principals’ component. We will mail an invitation letter and a paper questionnaire based on the 2017-18 NTPS mail procedures (see Attachment 2 and Attachment 3, respectively). Our survey will be mailed to a named school principal (if known) or to the generic title of “Principal.” Respondents will be asked to complete the 2017-18 NTPS Principals Questionnaire (approved under OMB# 1850-0598 v.18-21), into which will be inserted the different versions of confidentiality pledges that are being tested (on pages 2 and 28 of Attachment 3), and from which the contact information used for 2017-18 NTPS has been deleted.

After the initial mailing, nonrespondents will receive up to three additional contacts. Replacement questionnaires will be included in the third and fourth mailings. Respondents will be asked to complete and return the paper questionnaire using the enclosed pre-paid return envelope. A web response option will not be available due to budget constraints. We will not conduct telephone, email, or other follow-up operations.

A 3 x 2 factorial experiment will be embedded into the survey. Respondents will be randomly assigned to receive one of the six resulting treatments. Each treatment will contain one of three versions of the confidentiality pledge. These versions include a control treatment with the currently used confidentiality pledge, and two alternative versions – one which explicitly refers to DHS, and one which only refers to “Federal employees and contractors” and does not mention DHS. In addition, we will vary the inclusion of a statement in each of the three groups describing the penalties for government employees for disclosing survey responses. One half of the sample will receive the penalty statement, the other half will not. The treatment messages will be contained in the invitation letter (and in the Frequently Asked Questions section on the back of the letter), in each of the three reminder letters (see Attachment 2), in the introductory text within the questionnaire (Attachment 3, p.2), and in the questionnaire’s item 11-1 (Attachment 3, p.28). Each respondent will receive their assigned version of the confidentiality pledge throughout the mailings.

The sample for this experiment will be drawn from the NCES public school frame, using the 2015-16 Common Core of Data (CCD). In selecting a sample for this experiment, we will exclude schools that have been sampled for any of the regular production surveys conducted by NCES in the 2017-18 school year. We will also exclude districts which have refused to participate in the 2017-18 NTPS production survey. The sample for the experiment will be drawn from the cases remaining in the schools frame using a simple random selection procedure.

We will select a sample of 4,134 schools, which will be evenly divided among the six treatment groups. Table 1 shows the sample sizes for the control and treatment groups necessary to detect statistically significant differences of at least 5 percent, assuming an alpha of .05, power of 80 percent, and a response rate of 30 percent.

Table 1: Sample sizes for confidentiality pledge experiment treatment groups

	Penalty statement (N)	No penalty statement (N)	Treatment groups (Total N)
Confidentiality statement - control	689	689	1,378
Confidentiality statement - DHS	689	689	1,378
Confidentiality statement - No DHS	689	689	1,378
Total	2,067	2,067	4,134

The outcome of interest is the willingness of sample members to complete and send back their paper surveys, as measured by their response rate. A fully saturated two-way logit model will be fit to the response data. If an interaction effect is not detected, then the two main effects will be examined, a main effect for pledge variant and a main effect for the addition of penalty language.

3. Cost to Federal Government

The estimated cost to the Federal government to conduct this study is $179,546.

4. Participants

Participants in the survey will be school principals or other school staff who are assigned the task of completing the survey.

5. Response Burden

The burden per school for reading and potentially following up on the notification letter is estimated to average about 3 minutes. The burden for the NTPS Principal Questionnaire is estimated to average about 25 minutes. There are no differences in terms of the data items collected in each version of the questionnaire used for this experiment. Based on a projected response rate of 30%, we estimate 1,240 principals to complete the questionnaire. The estimated respondent burden for the NCES Confidentiality Pledges Mail Experiment is provided in Table 2.

Table 2. Estimates of respondent burden for the NCES Confidentiality Pledges Mail Experiment

Activity	Sample Size	Estimated Response Rate	Estimated Number of Respondents	Estimated Number of Responses	Average Burden Time per Respondent (Minutes)	Total Burden Hours
School notification letter	4,134	100%	4,134	4,134	3	207
Principal questionnaire*	4,134	30%	1,240	1,240	25	517
TOTAL	--	--	4,134	5,374	--	724

* Duplicative counts of individuals, not included in respondent totals.

6. Payment to Participants

Respondents will not receive any monetary incentive for their participation.

7. Confidentiality

Responses to our survey experiment are subject to the same confidentiality protections as the 2017-18 NTPS sample members (OMB# 1850-0598 v.18-21). Sample members will be provided with one of the confidentiality pledges listed in Attachment 1, depending on the experimental treatment to which they are randomly selected.

Analyses of responses will be completed collaboratively by Census Bureau and NCES personnel who are members of the project team. Reports of findings from the experiment will contain summary data and results from statistical analyses, and will not contain any information that can identify individual respondents or their schools.

8. Study and Reporting Schedule

The initial letter will be sent to sample schools in November 2017, upon OMB approval. Data collection will be concluded by May 2018, and the report of findings is expected to be finalized by September 2018.

Attachment 1: Confidentiality Pledges and Penalty Statements

Control version without penalty statement

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Control version with penalty statement

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). By law, any NCES employee or NCES contractor who willfully discloses any identifiable information about you or your school is subject to a jail term of up to 5 years, a fine of up to $250,000, or both.

Homeland Security version without penalty statement

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573). Electronic transmission of your information will be monitored for viruses, malware, and other threats by Homeland Security in accordance with the Cybersecurity Enhancement Act of 2015.

Homeland Security version with penalty statement

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573). By law, any NCES employee or NCES contractor who willfully discloses any identifiable information about you or your school is subject to a jail term of up to 5 years, a fine of up to $250,000, or both. Electronic transmission of your information will be monitored for viruses, malware, and other threats by Homeland Security in accordance with the Cybersecurity Enhancement Act of 2015.

Federal Employee Version without penalty statement

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573). Electronic transmission of your information will be monitored for viruses, malware, and other threats by Federal employees and contractors in accordance with the Cybersecurity Enhancement Act of 2015.

Federal Employee Version with penalty statement

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573). By law, any NCES employee or NCES contractor who willfully discloses any identifiable information about you or your school is subject to a jail term of up to 5 years, a fine of up to $250,000, or both. Electronic transmission of your information will be monitored for viruses, malware, and other threats by Federal employees and contractors in accordance with the Cybersecurity Enhancement Act of 2015.

Attachment 2: NTPS survey letters

Initial Letter (Two pages, front and back)

Date

Principal Name

School Name School Address

Dear <Principal NAME>:

You have been selected to complete the 2017-18 National Teacher and Principal Survey (NTPS) Principal Questionnaire. The NTPS is a primary source of information about what is happening in K–12 schools across the United States. The NTPS data are used by policy makers and researchers to understand the professional background, workload, and working conditions of principals in the United States.

Your participation in this important survey helps provide an accurate view of principals’ experiences as school administrators. Each response is vital to ensure reliable, nationally representative data. The NTPS is administered by the U.S. Census Bureau on behalf of the National Center for Education Statistics (NCES), the statistical agency within the U.S. Department of Education.

Please complete the questionnaire within one week and return it to the U.S. Census Bureau using the enclosed postage-paid envelope.

Please be assured that both the U.S. Department of Education and the U.S. Census Bureau follow strict procedures to protect the confidentiality of survey participants. <INSERT ASSIGNED PLEDGE HERE.> Your participation is voluntary, but your responses are necessary to make the results of this survey accurate and timely.

Thank you in advance for your participation in this important survey.

Sincerely,

Peggy G. Carr, Ph. D.

Acting Commissioner

National Center for Education Statistics (NCES)

U.S. Department of Education

What is the purpose of the NTPS?

The purpose of the National Teacher and Principal Survey (NTPS) is to provide policymakers and researchers with relevant and timely data on the characteristics and conditions of America’s public, charter, and private schools and the professionals who work in them. The data collected permit detailed analyses of the characteristics of schools, principals, teachers, and students. The data also link the NTPS components, which enables researchers to examine the relationships among these elements of the education system. For example, researchers can study teacher attrition using information not only provided by teachers, but also from their principals.

The NTPS has been designed with input from state and local education agencies, school administrators, teachers, education policy makers, and researchers through the numerous organizations representing these various data providers and data users. For more information about the NTPS, please visit our website at http://nces.ed.gov/surveys/ntps.

How are the NTPS data used?

The data from the NTPS are used by Congress, the U.S. Department of Education, state education agencies, public school districts, and education research organizations to:

1. Evaluate the effects of school workplace conditions, salaries, and training opportunities on the educational work force;

2. Assess school staffing practices and personnel policies; and

3. Aid in the Department of Education’s program planning in the areas of teacher shortage incentives, teaching policies, and teacher education.

Why is my school’s participation important?

This survey is one of the primary sources for information about K-12 schools in the U.S. Only a small percentage of schools are selected to participate; therefore, your participation is important for the success of the survey.

How long does it take to complete the NTPS Principal Questionnaire?

We estimate that the questionnaire will take approximately 25 minutes to complete.

Will my data remain confidential?

Please be assured that both the U.S. Department of Education and the U.S. Census Bureau follow strict procedures to protect the confidentiality of study participants. <INSERT ASSIGNED PLEDGE HERE.> Your participation is voluntary, but your responses are necessary to make the results of this study accurate and timely.

Who authorizes this survey?

The National Center for Education Statistics (NCES), within the U.S. Department of Education, is authorized to conduct this survey by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). The U.S. Census Bureau will administer this survey on behalf of NCES. The Office of Management and Budget (OMB) approved this survey. The OMB control number is 1850-0803 and the approval expiration date is 7/31/2019.

First Reminder Letter (one page)

Date

Principal Name

School Name School Address

Dear <PRINCIPAL NAME>:

We recently mailed you an invitation to complete the 2017-18 National Teacher and Principal Survey (NTPS) Principal Questionnaire. NTPS helps ensure that policy makers, universities, and researchers have a current and accurate picture of the education profession, including principals’ education and training, workloads, and working conditions across this country. If you have already completed and mailed back the survey, thank you. If you have not had the opportunity to complete your questionnaire yet, please complete it as soon as possible and return it using the postage-paid envelope enclosed with the initial letter.

The U.S. Department of Education and the U.S. Census Bureau follow strict procedures to protect your confidentiality. <INSERT ASSIGNED PLEDGE HERE.> Your participation is voluntary, but your responses are necessary to make the results of this survey accurate and timely.

The National Center for Education Statistics (NCES), within the U.S. Department of Education, is authorized to conduct this survey by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

Your participation in this important survey is greatly appreciated.

Sincerely,

Peggy G. Carr, Ph. D.

Acting Commissioner

National Center for Education Statistics (NCES)

U.S. Department of Education

Second Reminder Letter (one page)

Date

Principal Name

School Name School Address

Dear <PRINCIPAL NAME>:

A few weeks ago, we sent you a second invitation to complete the Principal Questionnaire for the 2017–18 National Teacher and Principal Survey (NTPS), a nationwide survey of elementary and secondary schools and the teachers and principals who staff them.

According to our records, you have not yet completed the survey. Your cooperation in this effort is greatly appreciated. This is your opportunity to participate in this important survey, which helps provide an accurate view of schools across this country.

If you have recently completed and mailed back the questionnaire, we want to thank you. If not, please complete the enclosed questionnaire, and return it to us in the postage-paid envelope as soon as possible.

The National Center for Education Statistics (NCES), within the U.S. Department of Education, is authorized to conduct this survey by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). <INSERT ASSIGNED PLEDGE HERE.>

Your participation is voluntary, but your responses are vital to ensure the results of this survey represent principals like you.

Thank you for your contribution to this important survey.

Sincerely,

Peggy G. Carr, Ph.D.

Acting Commissioner

National Center for Education Statistics (NCES)

U.S. Department of Education

Third Reminder Letter (one page)

Date

Principal Name

School Name School Address

Dear <PRINCIPAL NAME>:

Over the past two months, we have contacted you to ask for your participation in the 2017–18 National Teacher and Principal Survey (NTPS). We know that your time is valuable, but we ask you to share your experiences as a school administrator by completing the Principal Questionnaire. Your response is vital to ensure reliable, nationally representative data. Your responses are used by Congress and education research organizations to inform education policy.

If you have already completed the questionnaire, thank you for your assistance, and please disregard this letter. If you have not had the opportunity to complete the questionnaire yet, please complete the enclosed questionnaire as soon as possible.

The National Center for Education Statistics (NCES), within the U.S. Department of Education, is authorized to conduct this survey by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). Please be assured that both the U.S. Department of Education and the U.S. Census Bureau follow strict procedures to protect the confidentiality of survey participants. <INSERT ASSIGNED PLEDGE HERE.> Your participation is voluntary, but your responses are necessary to make the results of this survey accurate and timely.

Thank you for your time and consideration.

Sincerely,

Peggy G. Carr, Ph.D.

Acting Commissioner

National Center for Education Statistics (NCES)

U.S. Department of Education