Part B - 2021-22 TFS PFS v11

This request is to conduct data collection for the two follow-up surveys to the 2020-21 National Teacher and Principal Survey (NTPS) – the 2021-22 Teacher Follow-up Survey (TFS) and the 2021-22 Principal Follow-up Survey (PFS).

Section B.1 of this document describes the universe, sample design, and estimation details for the 2021-22 TFS and PFS. Section B.2 describes the data collection procedures for both surveys. Section B.3 discusses methods to secure cooperation and mitigate nonresponse. In particular, it describes methods used to improve response rates in the 2020-21 NTPS and how those methods will be used in its two follow-up surveys. Section B.4 describes recent developments in a long history of tests of methods and procedures to improve data quality. Section B.5 lists the names of those involved in the design of the study and the development of these materials.

B.1 Respondent Universe and Sample Design

B.1.1 Background on the 2020-21 NTPS Sample

The 2020-21 National Teacher and Principal Survey (2020-21 NTPS) sample was a two-stage stratified sample. The schools were sampled first, and then teachers were selected within each sampled school.

The NTPS school sample is a systematic probability proportionate to size (PPS) sample, where size is defined to be the square root of the full-time equivalent number of teachers at the school. Unlike its predecessor, the School and Staffing Survey (SASS), the NTPS does not stratify schools prior to sampling. However, some schools are oversampled based on their assigned domains. The public school oversampling domains are defined by charter status (charter/not charter), grade level (elementary/middle/high/combined), urbanicity (city/suburban/town/rural), and poverty status (> 75% of enrollment eligible for free or reduced price lunch/otherwise). The private school oversampling domains are defined by affiliation (11 levels), school span (Elementary/Secondary/Combined), school size (less than 100 students, or 100+ students), and Census division (the Mid-Atlantic states vs. other regions).

Public charter schools were oversampled at a rate 3.26 times their proportional sample allocation. Proportional allocation in this instance is defined as the domain’s sum of schools’ measures of size relative to the sum of measures of size of all schools in the sampling frame. Further oversampling applied to non-charter schools by grade level, whereby combined schools were oversampled at 2.56 times the elementary allocation, middle schools were oversampled at 1.30 times the elementary allocation, and high schools were oversampled at 1.35 times the elementary allocation. In addition, once the grade level sampling differential was applied, non-charter schools had their probabilities further altered by urbanicity as follows: city schools were oversampled at a rate of 1.05 times the proportional allocation (equal to the suburban sampling rate), town schools were oversampled at a rate 1.34 times the suburban rate, and rural schools were under sampled at a rate of 1.11 times the suburban rate.

In addition to oversampling by characteristic, schools in thirty (generally smaller, based on population size) states have relatively higher sampling rates to realize lower bounds on precision for these states and to enable the calculation of reliable key state estimates.

Along with the names of employed teachers and their email addresses, sampled schools were asked to provide the following descriptive characteristics of each teacher: teaching status, where options were either part-time or full-time; and subject matter taught, where teachers were classified as special education, general elementary, math, science, English/language arts, social studies, vocational/technical, or other.

The above information for each teacher in a selected NTPS school comprised the initial teacher sampling frame. The frame was also supplemented with information from vendor-purchased teacher roster files and from a clerical look-up operation conducted as part of NTPS where teacher lists were located on school and/or district websites. Within each sampled school, teachers were stratified by subject. The strata include math, science, English/language arts, social studies, and everything else. No oversampling was done for the teacher sample, beyond the oversampling inherited from their schools.

Teacher records within a school domain and teacher stratum were sorted by the teacher subject and the teacher line number code, which is a unique number assigned to identify the teacher within the list of keyed teachers. Within each teacher stratum in each school, teachers were selected systematically with equal probability.

Table B1 below shows a summary of the expected 2020-21 NTPS allocated sample sizes.

Table B1. Summary of the expected 2020-21 NTPS sample sizes

	School	Teacher
Traditional Public	8,660	43,460
Public Charter	1,260	5,790
Private	3,000	6,300
Total	12,920	55,550

B.1.2 2021-22 TFS Respondent Universe and Sample Design

The 2021-22 Teacher Follow-up Survey (2021-22 TFS) sample will be a subsample from the 2020-21 NTPS teacher sample. The 2021-22 TFS sample is a random subsample from the 2020-21 NTPS teacher sample, and, as such, a probability sample similar to 2020-21 NTPS.

A planned total of 10,273 teachers will be selected according to a stratified design similar to earlier TFS collections and based on information obtained from the NTPS schools about the status of the 2020-21 NTPS respondent teachers in the 2021-22 school year. The 2021-22 TFS sample will include former teachers (NTPS teachers who have left the teaching profession, or “Leavers”), current teachers (NTPS teachers who have remained in the teaching profession, or “Stayers”), and teachers who have left their 2020-21 school, but for whom there is no information available (the ‘unknown’ group). The current teachers group is further stratified into two groups: those who remained in the school they were teaching in 2020-21 (“Stayers”) and those who moved to different schools (“Movers”). This design provides analytic data on teachers who stay, move, or leave the teaching profession. The status for each TFS teacher is determined through a Teacher Status Form (TFS-1) mailed to schools in the fall of 2021. For the 2021-22 TFS, the Movers, Leavers, and Unknown teachers will be sampled with certainty, as will all Stayers in their first year.

Teachers will be further stratified by factors known about the teacher from the NTPS year, including sector (traditional public, public charter, and private), grade level (elementary, middle, and high)¹, experience level (1 year, 2-5 years, and 6+ years of teaching), and race/ethnicity (Hispanic, Black non-Hispanic, and other). Grade level is based on the grade level the teacher is teaching, but school grade level is used as the second choice if the teacher teaches in more than one grade level.

Within the explicit strata, there will be systematic sampling with a designated sort order. The sort variables include measure of size (teacher base weight), teacher subject, Census region, urbanicity, school enrollment, and for private school teachers, type of private school.

Teachers will be selected within each stratum using a probability proportional to size sampling procedure. The measure of size will be the 2020-21 NTPS final teacher weight. This essentially ‘reverses’ the probability of selection of the teacher in NTPS, and results in a self-weighting sample within each teacher stratum. This will maximize the efficiency of the TFS sample.

The table below presents target sample sizes for important teacher categories. Note that charter school and private school teachers are oversampled relative to traditional public school teachers. This will be done to reduce the coefficient of variation (CV) for charter school estimates to approximately 20 percent. Movers, leavers, and teachers with unknown status are all selected, which makes them overrepresented in the sample relative to other teachers. This is done to increase the precision for analyses of these important subgroups. The breakdown of stayers, movers, leavers, and ‘unknown’ teachers in public, charter, and private schools is as shown in Table B2.

Table B2. Target Distribution of teacher types across school types

Status	Sector
Status	Public	Charter	Private	Total
Stayer	2,810 (0.093)	538 (0.264)	1,026 (0.269)	4,347
Mover	2,214	205	246	2,665
Leaver	1,434	107	278	1,819
Unknown	1,029	182	204	1,415
Total	7,487	1,032	1,754	10,273

Note: The number in parentheses is the estimated sampling rate for each Stayer cell.

Within these target sample sizes, Hispanic and Black Non-Hispanic stayers are oversampled by 60 percent to optimize the reliability of their comparison with other stayers (Movers, Leavers, and Unknown are already sampled with certainty). The target sample sizes at the status/sector/race level are as seen in Table B3.

Using these target sample sizes, the sample is allocated to the sampling strata, which are defined by the teacher’s sector, status, experience, grade, and race categories. This is done proportional to the cumulative teacher measure of size within each stratum. The teacher measure of size will be the 2020-21 NTPS teacher base weight.

Table B3. Distribution of teacher types across school types

Status	Sector									Total
	Public			Charter			Private
	Hispanic	Black Non-Hispanic	Other	Hispanic	Black Non-Hispanic	Other	Hispanic	Black Non-Hispanic	Other
Stayer	258	252	2,300	106	110	322	209	166	651	4,374
Mover	116	126	1,972	22	20	163	14	20	212	2,665
Leaver	44	67	1,323	8	11	88	10	10	258	1,819
Unknown	79	89	861	22	28	132	19	20	165	1,415
Total	497	534	6,456	158	169	705	284	216	1,286	10,273

B.1.3 2021-22 PFS Respondent Universe and Sample Design

The sampling frame for the 2021-22 PFS consists of all the traditional public and public charter school principals who completed a Principal Questionnaire and all private school principals who completed a Private School Principal Questionnaire during the 2020-21 NTPS. Any sampled NTPS principal who did not complete their questionnaire or was otherwise found to be out of scope for NTPS will not be included in the PFS frame. The 2021-22 PFS sample will include approximately 6,700 public and public charter school principals and 1,750 private school principals.

The 2016-17 PFS included a sample of 5,700 principals from public and charter schools. The large discrepancy in sample size between the 2016-17 and 2021-22 administrations of the PFS is due to the fact that the 2015-16 NTPS included fewer sampled schools than the 2020-21 NTPS, because the 2015-16 NTPS sample included public school principals only and produced estimates at the national rather than state level.

B.2 Procedures for the Collection of Information

B.2.1 2021-22 TFS Procedures for the Collection of Information

B.2.1.1 Teacher Status Form (TFS-1)

The TFS-1 will be mailed to all schools in which a teacher completed a Teacher Questionnaire during the 2020-21 administration of NTPS. A knowledgeable person at the school, such as the principal, will be asked to complete the TFS-1 by indicating the current teaching status of each teacher listed on the form. These data will be used to stratify the teachers for 2021-22 TFS sampling into groups of “stayers” (still teaching at the same school), “movers” (still teaching, but at a different school), or “leavers” (no longer teaching).

TFS was designed from its inception to sample only from respondents to SASS/NTPS in the prior year. It is possible to sample teachers for TFS from the pool of teachers initially selected for NTPS rather than only from those who responded to NTPS. By not sampling teachers from the nonresponse stratum, potential biases may be introduced into TFS. When considering unit nonresponse for SASS, there was no evidence to point to a substantial bias in teacher response rates.

The main purpose of TFS is to measure various characteristics collected on NTPS of teachers who move, leave, or stay in the teaching profession the following year. If NTPS nonrespondents were to be contacted for the TFS, there are several methodological problems that would need to be solved: 1) there would be no prior year data, so all of the NTPS teacher questionnaire items would have to be asked retrospectively of the teacher before the TFS could be administered; 2) if the teacher was determined to be out-of-scope for NTPS, then there would be no TFS interview and a sampling adjustment would be required; and 3) if the teacher was in-scope for NTPS but out-of-scope for TFS (if the teacher moves out of the country or dies), then that would also require some adjustment of the sampling weights. Logistically, trying to combine prior-year NTPS interviewing with current-year TFS interviewing is a complication that would delay the field data collection because it would likely take longer to complete the nonresponding NTPS cases. For these reasons, TFS teachers will be sampled from responding NTPS teachers only. A response rate of approximately 99 percent can be expected from the TFS-1 based on the prior TFS data collection cycle.

B.2.1.2 Questionnaires for Former and Current Teachers (TFS-2 and TFS-3)

Using the teacher statuses provided by schools on the TFS-1, Census Bureau staff will select the sample of teachers for the 2021-22 TFS and prepare materials for data collection.

In the 2020-21 NTPS, teachers were asked to provide the following information:

Name

Street address

Cell phone number

Home phone number

Most convenient day(s) to reach her/him

Work email address

Home email address

Prior to the start of data collection, Census staff will conduct research to obtain contact information (e.g., home address, email address, and/or telephone number) for teachers who provided incomplete or no contact information in the NTPS interview. Staff will first use any contact information provided (e.g., telephone number, partial address, contact person’s information) from the NTPS questionnaire to attempt to complete the address of the sample member. If no contact information was provided, staff will use sources such as the white or yellow pages, Internet searches, and other address and phone number databases.

Data collection for the Questionnaire for Former Teachers (TFS-2) and the Questionnaire for Current Teachers (TFS-3) will begin in January 2022. The first contact with sample members will be through an initial email invitation to participate (unless a valid email address is unavailable) with a link to the survey and their log in credentials. If home addresses were not provided on the 2020-21 NTPS teacher questionnaire or cannot be obtained, the NTPS school address will be used as the address. Shortly after this email, an initial contact letter will also be sent by mail to nonrespondents inviting them to participate in the survey and providing credentials for them to log on and complete the appropriate web-based instrument. All teachers will also receive a $5 or $10 cash incentive in this initial mailed package. Note that the monetary ($5 or $10 cash) incentives for teachers will be adhered to a piece of yellow cardstock using removable sticky glue and will be inserted with the letter. The cardstock will be ½ sheet rather than a full sheet and will include text thanking them for their participation in the study. Adhering the cash to the cardstock insert will (1) increase the weight of the envelope, making it feel more “substantial” and important; (2) prevent the money from free-floating inside the envelope; and (3) help ensure that the respondent notices the cash. Each letter and email will be customized and will provide the sampled teacher with login information to access the web-based instrument.

In late January, Census will send nonrespondents a first email reminder, followed by a second teacher mailout in early February. For a subsample of teachers, this package will also include an experimental infographic. Refer to section B.4.2 for additional information about testing the inclusion of an infographic in TFS packages. A second reminder will be sent by email only in mid-February.

The third email reminder will be sent to nonrespondents in early March, followed by a third mailout to nonrespondents in mid-March. A fourth reminder by email only will be sent in late March, followed by a fifth and sixth reminder by email. The fifth and sixth reminder emails will be sent around the same time as the fourth and fifth mailed packages, which will also include a paper copy of the applicable TFS questionnaire. Data collection will end in early July 2022. Throughout data collection, research will be conducted, as needed, to find current addresses and emails for sampled teachers whose letters and emails are returned as undeliverable.

In addition, ongoing support will be provided for sampled teachers to whom the incorrect paper questionnaire type was mailed (e.g., a mover who was incorrectly classified as a leaver and received a TFS-2 questionnaire as a result). When these cases are discovered, either because the teacher was Amish or Mennonite or in response to the late mailing(s) of the paper questionnaires, Census will send the correct form for completion.

From February through June, outbound telephone calls to remind teachers who have not yet responded to the web-based collection to complete their form. While the call is intended to encourage teachers to log on and respond to the survey themselves, interviewers will be able to administer it over the phone at the request of the respondent. Throughout the data collection period, respondents who request a paper questionnaire will be mailed one.

The 2021-22 TFS will include an experiment to test different methods of texting survey participants as a means of exploring the use of text messages as a new data collection mode. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS teacher questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.

To explore using test messaging as a contact method for TFS, the experiment will include three treatment groups:

Replace the second and third mailouts (web invitation letters) with text message contacts that include the link to complete the TFS online
Include a text message contact that includes a link to complete the TFS online after the fourth mailout (first mailed paper questionnaire), and replace the fifth mailout (second mailed paper questionnaire) with a text message contact that includes a link to complete the TFS online
Include an interactive questions text message contact after the fourth mailout (first mailed paper questionnaire) and replace the fifth mailout (second mailed paper questionnaire) with an interactive questions text message contact

For groups one and two, the text message will invite the teacher to complete the TFS questionnaire online by providing the link to the web instrument in the text message. For group three, questions about current teaching status will be texted directly to each sampled teacher and his or her response will be collected via their text messaged response. These interactive texts will follow the same general skip pattern as the web instrument, where subsequent texts will ask follow-up questions based on the teacher’s responses. Results from this experiment will help determine whether a respondent is more likely to respond via a texted survey web link earlier in data collection versus later in data collection, as well as whether a respondent is more likely to respond via a texted survey web link or directly in an interactive text exchange later in data collection.

Outside of this experiment, text messages will also be used to encourage response from non-responding teachers. More specifically, toward the end of data collection and after all text message experiment activities have concluded, non-responding, consenting teachers from the control group will be sent text messages in two stages. The first stage will include sending a link to complete the TFS online, while the second will include sending the interactive questions. Non-responding teachers from treatment groups one and two will be sent the interactive questions, as well, in a final attempt to obtain their teaching status prior to the end of data collection. For additional details about testing the use of text messaging in the 2021-22 TFS, refer to section B.4.1 of this document.

A response rate of approximately 75 percent (base-weighted) can be expected from the leavers after follow up, and a response rate of about 81 percent (base-weighted) can be expected from stayers and movers after follow-up, based on the prior administration of TFS. Table B4 shows weighted unit response rates in the last administration of TFS, TFS 2012-13.

Table B4. Weighted unit and overall response rates of TFS teachers, using initial base weight, by sector and teaching status: 2012–13

Sector and teaching status	Weighted unit response rate	Weighted overall unit response rate
Public	80.7	49.9
Current teacher	81.3	50.3
Stayer	81.6	50.5
Mover	77.9	48.2
Former teacher	73.8	45.6
Private	79.3	39.7

B.2.2 2021-22 PFS Procedures for the Collection of Information

In January 2022, Census Bureau staff will mail the PFS-1(A/B)^² to all schools to collect the current occupational status of the 2020-21 principal. Non-responding schools will be sent a reminder letter along with a second copy of the PFS-1 (A/B) and reminder email approximately three weeks after the initial mailing. Those schools that are still non-responsive after the second mailout and reminder email will receive a telephone call during which telephone center staff will follow-up by telephone to collect the occupational status of the 2020-21 principal. The telephone center staff will first ask school office staff for the occupational status of the 2020-21 principal. If the office staff is unable to provide the information, telephone center staff will ask to speak with the current school principal.

For non-responding schools or schools that were unable to report the principal’s status, Census Bureau staff will mail a modified version of the Principal Status Form (PFS-1(C/D)³) to the 2020-21 principal at his or her home address (if reported in NTPS). The modifications tailor the form to the previous years’ principal rather than the school staff. This mailing will begin in March 2022, and an email will be sent a few days following the mailing, alerting the principal to the mailing and reminding them to complete their PFS. An alternate version of this email will also be sent experimentally to a smaller group of principals and will contain a link to complete the PFS online via a Qualtrics web-based instrument. All non-responding principals will be sent a follow-up mailing and email (either the standard or alternate Qualtrics version, depending on their treatment group) approximately 2 weeks after the initial mailing and email. Telephone center staff will follow-up by telephone to collect the occupational status of the non-responding principal beginning two weeks after the reminder mailing and email are sent.

The 2021-22 PFS will also include an experiment to test different methods of texting survey participants as a means of exploring the use of text messages as a new data collection mode. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number. All text message contacts for the survey will be an interactive exchange with the principal.

To explore offering a web-based response option for PFS and to explore using text messaging as a contact method for PFS, the PFS data collection will include two treatment groups:

Offering an interactive questions text message contact to complete the survey concurrent with both principal-level mailouts.
Sending an email containing a link to complete the PFS online via a Qualtrics web-based instrument concurrent with both principal-level mailouts.

The experiment will help determine whether offering principals a web-based response option helps to boost PFS response and whether survey participants would be willing to respond to a survey via text, generally. In addition, results from this experiment will help determine whether the text serves as a reminder, prompting respondents to return their paper questionnaire. For additional details about testing a web-based response option and testing the use of text messaging in the 2021-22 PFS, refer to section B.4.2 of this document.

A response rate in the approximate range of 90-95 percent is expected based on the prior administration of PFS.

B.3 Methods for Maximizing Response

This section describes the methods that NCES will use to secure cooperation, maximize response rates, and deal with nonresponse for the 2021-22 NTPS follow-up surveys. The entire survey process, starting with securing research cooperation from key groups and individual sample members and continuing throughout the distribution and collection of individual questionnaires, is designed to increase survey response rates. In addition, NCES believes that the following endorsements, stressing the survey’s importance in communication materials, and extensive follow-up with nonrespondents will further facilitate the overall success of the survey and enhance response rates.

B.3.1 Endorsements from Key Public and Private School Groups

The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. Endorsements are viewed as a critical factor in soliciting cooperation from state and local education officials.

NCES will seek endorsements for the 2021-22 TFS from the following organizations or agencies:

Agudath Israel of America/Lefkowitz Leadership Initiative

American Association of Christian Schools

American Association of School Administrators

American Federation of Teachers

American Montessori Society

Association for Middle Level Education

Association for Middle Level Education (formerly National Middle School Association)

Association for Supervision and Curriculum Development

Association Montessori International

Association of American Educators

Association of Christian Schools International

Association of Christian Teachers and Schools

Association of Classical Christian Schools

Association of Military Colleges and Schools

Association of Waldorf Schools of North America

Christian Schools International

Council for American Private Education

Council of Chief State School Officers

Council of Islamic Schools of North America

Council of the Great City Schools

Evangelical Lutheran Church in America

Friends Council on Education

General Conference of Seventh-Day Adventists

Islamic School League of America

Jesuit Schools Network

Jewish Education Services of North America

Lutheran Church-Missouri Synod

National Association of Elementary School Principals

National Association of Episcopal Schools

National Association of Independent Schools

National Association of Private Special Education Centers

National Association of Secondary School Principals

National Catholic Educational Association

National Christian School Association

National Coalition of Girls’ Schools

National Council for Private School Accreditation

National Education Association

National Independent Private School Association

National Parent Teacher Association

Office of Education, General Conference of Seventh Day Adventists

Oral Roberts University Educational Fellowship

Prizmah: Center for Jewish Day Schools

RAVSAK: Jewish Community Day School Network

Southern Baptist Association of Christian Schools

The Association of Boarding Schools

The Jewish Education Project

The School Superintendents Association

Torah Umesorah National Society for Hebrew Day Schools

United States Conference of Catholic Bishops

Wisconsin Evangelical Lutheran Synod

NCES will seek endorsements for the 2021-22 PFS from the following organizations or agencies:

American Association of School Administrators

American Federation of Teachers

National Association of Elementary School Principals

National Association of Secondary School Principals

B.3.2 Stressing the Importance of the Survey and the Respondents’ Participation in it

Official letters will be used to motivate respondents to return surveys. Respondent letters for both the 2021-22 TFS and PFS will be sent by the U.S. Census Bureau and signed by the NCES Commissioner. Communications in the form of both letters and emails will be personalized with the teacher and principals’ names, which is expected to have positive effects on the survey response rates.

B.3.3 Offering a cash incentive to current and former teachers

The 2004-05 TFS used monetary incentives to increase overall response and internet response when both mail and internet choices were offered. The results showed that the overall response of the incentive groups significantly exceeded that of the non-incentive groups.⁴ Additionally, the results of the use of teacher incentives for the 2017-18 NTPS indicated that the teacher incentive led to significant increases in the response rate for both public and private school teachers.⁵ In addition, the average number of days to complete the questionnaire was significantly lower for public school teachers that received the incentive. Finally, the incentive helped increase the overall sample balance for teachers in both public and private schools.

Due to these favorable results, the 2020-21 NTPS included the use of a cash incentives in the initial mailed packages to a subsample of early sampled teachers and as a promised incentive, contingent upon a teacher completing his or her teacher questionnaire, for teachers sampled in December and later. Mid-data collection, a switch to promised incentives was made in an attempt to encouraging teachers to respond to the survey, given the challenges posed during the 2020-21 school year by the coronavirus pandemic. As such, the 2021-22 TFS will include a cash incentive for sampled teachers. The incentives included in the TFS will not be offered in an experimental manner and will instead be offered to all sampled current and former teachers.

Data on teacher retention and attrition are in high demand from researchers and policymakers, but TFS data have not been released since 2012-13. The TFS was last fielded in 2012-13, after the 2011-12 SASS. During the 2012-13 TFS, response rates from private school teachers was problematically low, and NCES was unable to publish a TFS private school teacher data file. In 2016-17, after the 2015-16 NTPS, the overall unit response rate of the TFS 2016-17 was likely to be under 50 percent and therefore the entire collection was cancelled. In an effort to increase response rates for the 2021-22 cycle, a contingency incentive may also be used in later TFS mailings to combat low response, as identified in weekly status reports. A similar contingency incentive was included with the 2017-18 NTPS and led to a boost in response rate for teacher who received this incentive.⁶

During week 11 of data collection (3/22/2022), we will evaluate the response rates for the TFS-2 (former teachers) and TFS-3 (current teachers) and determine whether a contingency incentive is needed. Specifically, if the TFS-2 response rate is less than 58%, those teachers will be eligible to receive a contingency incentive. If the TFS-3 response rate is less than 64.5%, those teachers will be eligible to receive a contingency incentive. Teachers in the early text web link group will also be eligible to receive a contingency incentive. Depending on whether response rates are struggling overall or for certain types of schools, the contingency plan may be exercised for teachers in all types of schools or only teachers sampled from schools with particular characteristics (e.g., priority schools⁷). The amount of the contingency incentive will be $10. This will be a one-time, pre-paid cash incentive to be mailed directly to the responding teacher in place of the fourth mailout.

B.3.4 Extensive Follow-up (by mail, email, and telephone) of Non-Respondents

For the 2021-22 TFS, the Census Bureau will use a variety of techniques to increase response levels including:

(a) Using mixed survey modes— self-administered web-based, telephone interviewer-assisted web-based, and self-administered mail instruments, as needed, to maximize response levels.

(b) Allocating adequate time and resources to respondent tracking efforts to ensure that a high percentage of movers and leavers are successfully located and surveyed. A variety of techniques will be employed to locate survey respondents. Potential tracking sources include: (a) leads provided by school principals or their designees; (b) clerical operations using electronic databases and internet searches; and (c) post office for possible forwarding addresses.

Good questionnaire design techniques will be employed to minimize item non-response. All completed questionnaires from TFS 2012-13 were analyzed to determine which items had the highest levels of item nonresponse. This information guided NCES and the Census Bureau in reviewing the clarity of item wording, definitions, and instructions, and as such, revisions were made to TFS content for the TFS 2016-17. The TFS 2016-17 was approved by OMB (#1850-0934 v1); however, the data collection was cancelled due to lower than anticipated response rates for subdomains of 2015-16 NTPS teachers. Part C of this submission details the minor changes between the proposed TFS instruments for 2021-22 and those approved by OMB for the TFS 2016-17.

To permit sufficient time to locate sampled members who left their teaching position in the previous year and thorough follow-up of non-respondents, the TFS teacher data collection will begin in January 2022, allowing six months for all data collection activities related to teachers.

For the PFS, the Census Bureau will also utilize mixed survey modes – self-administered mail instruments, a web-based Qualtrics instrument (for a subsample of principals), and telephone, as needed, to maximize response levels. Non-respondents will receive at least two reminders. Given the brevity of the instrument, we anticipate response rates at or above 90% with this approach, as with past cycles of the survey.

B.4 Tests of Procedures and Methods

NCES is currently considering options for tests of methods, materials, and procedures to be conducted as part of the 2021-22 NTPS Follow-up Surveys, with the goal to increase response in the 2020-21 collection. Each test is described in detail below.

B.4.1 Testing the use of text messages in TFS

For 2021-22 data collection cycle of the TFS, NCES will explore the use of text messaging as a new data collection mode by including an experiment to test different methods of texting survey participants. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS teacher questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.

To explore using test messaging as a contact method for TFS, the experiment will include three treatment groups:

Replace the second and third web invitation letter mailouts with text message contacts that include the link to complete the TFS online
Include a text message contact that includes a link to complete the TFS online after the fourth mailout (the first paper questionnaire mailing), and replace the fifth mailout with a text message contact that includes a link to complete the TFS online
Include an interactive questions text message contact after the fourth mailout and replace the fifth mailout with an interactive questions text message contact

Only teachers who opt-in to receive text messages are eligible for inclusion in the text messaging experiment. The estimated opt-in rate for public and private school teachers is 60%. The treatment for each sampled teacher will be randomly assigned at the school-level at the time of teacher sampling, prior to the start of teacher data collection. Any school where at least one teacher consented to receiving text messages in the experiment is eligible to be included. The schools will be sorted by school size and other characteristics related to response. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. Only schools where all teachers did not consent to receiving text messages would be considered ineligible for the experiment. If a school is selected to receive text messages, all sampled teachers who had opted-in would receive the same text message treatment, and any sampled teachers who did not consent would not receive any text messages as part of the experiment. The teachers in the sampled school who opted out of text messaging would receive the traditional mail contacts. Nonrespondent teachers may not be substituted with other sampled teachers. The experimental contact plan and expected sample sizes are displayed in Exhibit 1.

Exhibit 1: 2021-22 TFS Text Message Experimental Contact Strategy and Sample Sizes

Experimental Treatment and Sample Size	Planned Date	Concurrent with Text 1	1/10-1/12	1/28-1/31	2/16	3/07-3/08	3/31	4/15-4/19	5/20-5/23	6/10
Standard/Control Group 1,278 public school teachers 263 private school teachers	Mail Activity		1st Mail (W)	2nd Mail (W)		3rd Mail (W)		4th Mail (Q)	5th Mail (Q)
	Email Activity		Initial	Reminder 1	Reminder 2	Reminder 3	Reminder 4	Reminder 5	Reminder 6
	Telephone Activity				Phone Reminder				Phone Follow-Up
	Texting Activity

Early Texting - Web Link 1,278 public school teachers 263 private school teachers	Mail Activity		1st Mail (W)					2nd Mail (Q)	3rd Mail (Q)
	Email Activity		Initial	Reminder 1	Reminder 2	Reminder 3	Reminder 4	Reminder 5	Reminder 6
	Telephone Activity				Phone Reminder				Phone Follow-Up
	Texting Activity	Welcome Text	Text 1	Text 2		Text 3

Late Texting - Web Link 1,278 public school teachers 263 private school teachers	Mail Activity		1st Mail (W)	2nd Mail (W)		3rd Mail (W)		4th Mail (Q)
	Email Activity		Initial	Reminder 1	Reminder 2	Reminder 3	Reminder 4	Reminder 5	Reminder 6
	Telephone Activity				Phone Reminder				Phone Follow-Up
	Texting Activity	Welcome Text						Text 1⁹	Text 2

Late Texting – Questions⁸ 1,278 public school teachers 263 private school teachers	Mail Activity		1st Mail (W)	2nd Mail (W)		3rd Mail (W)		4th Mail (Q)
	Email Activity		Initial	Reminder 1	Reminder 2	Reminder 3	Reminder 4	Reminder 5	Reminder 6
	Telephone Activity				Phone Reminder				Phone Follow-Up
	Texting Activity	Welcome Text						Text 1⁸	Text 2

Following data collection, analyses will be conducted at the treatment level (experimental group vs. control). The analyses examined include:

Response rate;
R-Indicators;
Average number of contacts; and
Days to respond.

The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the text message(s) had on a case’s likelihood to respond, given that case’s unique characteristics.

Given the project sample sizes in the table above, a statistically significant difference between the control group (no text messages) and any of the other three treatment groups will be found if the response rates between the two groups differ by:

8.63% for public school teachers (TFS)
18.48% for private school teachers (TFS)

R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.

The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of contact strategy over another. If cases within the experimental group receiving text messages respond in a more timely fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, to be spread across fewer cases.

Outside of this experiment, text messages will also be used to encourage response from non-responding teachers. More specifically, toward the end of data collection and after all experiment activities have concluded, non-responding, consenting teachers from the control group will be sent text messages in two stages. The first stage will include sending a link to complete the TFS online, while the second will include sending the interactive questions. Non-responding teachers from treatment groups one and two will be sent the interactive questions, as well, in a final attempt to obtain their teaching status.

B.4.2 Testing the use of a web-based instrument and text messages in PFS

For the 2021-22 data collection cycle of the PFS, NCES will explore the use of text messaging as a new data collection mode by including an experiment to test different methods of texting survey participants. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.

All text message contacts for the PFS will be an interactive exchange with the principal. PFS data collection begins with contacts to the principal at the NTPS school and then shifts to contacts with the sampled principals directly using the personal contact information they provided on the NTPS if the school is not responsive.

Additionally, the 2021-22 data collection of PFS will explore the option of offering principals the option to complete the PFS online via a Qualtrics web-based instrument. This response mode was added in response to lower than expected response (when compared to previous data collection cycles of PFS) from schools during the 2021-22 PFS school-based data collection as means of boosting response rates.

To explore offering a web-based response option for PFS and to explore using text messaging as a contact method for PFS, the PFS data collection will include two treatment groups:

Offering an interactive questions text message contact to complete the survey concurrent with both principal-level mailouts.
Sending an email containing a link to complete the PFS online via a Qualtrics web-based instrument concurrent with both principal-level mailouts.

Only principals who opt-in to receive text messages are eligible for inclusion in the text messaging experiment. The estimated opt-in rate for public and private school principals is 60%. The treatment for each principal will be randomly assigned prior to the start of PFS data collection. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. The experimental contact plan and sample sizes are displayed in Exhibit 2.

Exhibit 2: 2021-22 PFS Experimental Contact Strategy and Sample Sizes

Experimental Treatment	Planned Date	School-level Contacts (PFS-1A/B)			Principal-level Contacts (PFS-1C/D)
Experimental Treatment	Planned Date	1/13/22	2/3/22	2/14/22-3/4/22	Concurrent with Text 1	3/16/22, 3/17/22, 3/22/22,	3/29/22, 3/31/22, 4/5/22	4/18/22 - 5/13/22
Standard/Control Group 1,322 public school principals 342 private school principals	Mail Activity	1st Mail	2nd Mail			1st Mail	Reminder Mail
	Email Activity					Reminder 1	Reminder 2
	Telephone Activity			Telephone FU				Telephone FU
	Texting Activity


Mail & Texting Treatment - Status question to determine attrition 1,322 public school principals 342 private school principals	Mail Activity	1st Mail	2nd Mail			1st Mail	Reminder Mail
	Email Activity					Reminder 1	Reminder 2
	Telephone Activity			Telephone FU				Telephone FU
	Texting Activity				Welcome Text	Text 1	Text 2

Web Treatment 1,322 public school principals 342 private school principals	Mail Activity	1st Mail	2nd Mail			1st Mail	Reminder Mail
	Email Activity					Initial Email (with link)	Reminder Email (with link)
	Telephone Activity			Telephone FU				Telephone FU
	Texting Activity

Following data collection, analyses will be conducted at the treatment level (experimental group vs. control). The analyses examined include:

Response rate;
R-Indicators;
Average number of contacts; and
Days to respond.

Given the project sample sizes in the table above, a statistically significant difference between the control group (no text messages) and any of the other two treatment groups will be found if the response rates between the two groups differ by:

6.26% for public school principals (PFS)
12.39% for private school principals (PFS)

B.4.3 Testing the inclusion of an infographic in TFS packages

Previous research has indicated that personalizing outreach materials can increase response rates, which has been used to inform procedures for communication materials (e.g., letters, emails) for past cycles of the NTPS, as well as the upcoming 2021-22 TFS and PFS collections.⁹^,¹⁰^,¹¹By and large, however, for these and other survey collections, generalized facts and figures have been included amongst materials in an effort to motivate sample participants to respond to the survey.

An experiment to include one teacher infographic, either a personalized infographic or a general infographic, in the second mailout package will be included in the 2021-22 TFS. Both types of infographics will present information with non-research audiences in mind, and the personalized infographics will provide facts from the NTPS that are more tailored to the individual sampled teacher versus the general teacher population. Since the 2021-22 TFS sample is drawn from the 2020-21 NTPS, and therefore data regarding the sampled teachers’ main teaching assignment are available prior to the beginning of the TFS, this experiment is a good fit for the TFS collection. By testing both general and the personalized infographics, we can also consider the value add of including general infographics to future NTPS mailed packages, since personalized teacher data is less available at that stage.

This experiment aims to address the following research questions:

Does the inclusion of any infographic affect response?
Does the inclusion of a personalized/targeted infographic affect response?
Does the inclusion of a general infographic affect response?

The general infographics will be developed by school classification. As a result, there will be two different variations of the general infographic:

Public school stayer/mover/leaver
Private school stayer/mover/leaver

The personalized infographics will be developed around both school classification and the 11 main teaching assignments that a teacher could select on their completed 2020-21 NTPS Teacher Questionnaire. In order for infographics to be truly personalized, there will also be different infographics for stayers/movers versus leavers. As a result, there will be 24 different variations of the personalized/targeted infographic:

Public school stayers/movers, by each of the 11 main teaching assignments
Private school stayers/movers, by each of the 11 main teaching assignments
Public school leavers
Private school leavers

The infographics will include three parts: (top) introductory text, (middle) statistics from the 2017-18 NTPS, and (bottom) call to action language.

The introductory text reminds participants of their participation in last year’s NTPS, informs them of the purpose of the TFS collection, and tells them how they can help make the data collection successful. This language will remain the same for all infographic conditions, and only minor personalization changes will be made for personalized leavers infographics (i.e., public school leavers and private school leavers) to address the fact that these are former teachers instead of current teachers.

The statistics are drawn from the 2017-18 NTPS public or private teacher data file, depending on the school type of the sampled participant. For the general infographics and the personalized stayer/mover infographics, the same items will be included. The statistics shared will be based on the targeted subgroup (i.e., for the public school and private school general infographics, or personalized for each of the two variations listed above by the eleven main teaching assignments). The personalized Leavers infographics (i.e., private school leavers and public school leavers) will contain the same items as one another, and differ slightly from the other personalized and general infographics, as a reflection of the fact that teachers who left the field and those who remained in the profession may be interested in different information.

The call to action language will remind participants how the TFS data are used and why it’s important that their experience be included in the survey results. Similarly to the introductory language, this language will remain the same for all infographic conditions, and only minor personalization changes will be made for leavers.

The treatment for each sampled teacher will be randomly assigned at the school-level at the time of teacher sampling, prior to the start of teacher data collection. The schools will be sorted by school size and other characteristics related to response. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. The experimental treatments and expected sample sizes are displayed in Exhibit 3.

Exhibit 3: 2021-22 TFS Infographic Experiment Treatments and Sample Sizes

Conditions

Treatment

Sample Size

Number of unique infographics made

Control

No infographic included in the second teacher mail-out

1,000 public school teachers

585 private school teachers

0 infographics

Condition 1

One general infographic included in the second teacher mail-out

(Appendix A, p.6)

3,760 public school teachers

585 private school teachers

1 infographic for public school teachers

1 infographic for private school teachers

Condition 2

One personalized/targeted infographic included in the second teacher mail-out

(Appendix A, p.7-8)

3,760 public school teachers

585 private school teachers

24 infographics

Public School Teachers

Stayer/Mover

11 infographics (one for each NTPS main assignment field)

Example: “public school mathematics teacher”, “public school foreign languages teacher”

Leaver

1 infographic

Example: “Former public school teacher”

Private School Teachers

Stayer/Mover

11 infographics (one for each NTPS main assignment field)

Example: “private school mathematics teacher”, “private school foreign languages teacher”

Leaver

infographic

Example: “Former private school teacher”

Following data collection, analyses will be conducted at the treatment level (experimental group vs. control). The analyses examined include:

Response rate;
R-Indicators;
Average number of contacts; and
Days to respond.

The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the infographic had on a case’s likelihood to respond, given that case’s unique characteristics.

Given the project sample sizes in the table above, a statistically significant difference between the control group (no infographic) and one of the treatments will be found if the response rates between the two groups differ by 7.76% for public school teachers and 12.40% for private school teachers. A statistically significant difference between the two treatments will be found if the response rates differ by 5.03% for public school teachers and 10.74% for private school teachers.

The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of contact strategy over another. If cases within the experimental group receiving infographics respond in a more timely fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, to be spread across fewer cases.

B.5 Reviewing Statisticians

Aaron Gilary and Alfred Meier of the U.S. Census Bureau, Jim Green of Westat, and Andy Zukerberg, Maura Spiegelman, and Julia Merlin of NCES reviewed and approved the TFS and PFS sample design and related matters for statistical quality, feasibility, and suitability to the overall objectives of the survey.

1 Elementary: teachers who taught elementary school students (any grade K-8, but at least one of the grades K-4) in the 2020-21 school year; Middle: teachers who taught middle school students (grades 5-8 exclusively) in the 2020-21 school year; High: teachers who taught high school students (any grade 7-12, but at least one of grades 9-12) in the 2020-21 school year.

2 The “A” version is for public schools; the “B” version is for private schools.

3 The “C” version is for public schools; the “D” version is for private schools.

4Tourkin, S., Parmer, R., Cox, S., & Zukerberg, A. (2005). (Inter) Net Gain? Experiments to Increase Response. Paper presented at the 60^th Annual Conference of the American Association for Public Opinion Research (AAPOR), Miami Beach, FL, May.

5Zukerberg, A., Zotti, A., & Cox, S. (2019). Better Late Than Never? The Use of An Adaptive Incentive with Nonrespondents. Proceedings of the American Association for Public Opinion Research (AAPOR) Annual Conference, Toronto, Canada, May.

6 Zukerberg, A., Zotti, A., & Cox, S. (2019 Better Late Than Never? The Use of An Adaptive Incentive with Nonrespondents. Proceedings of the American Association for Public Opinion Research Annual Conference.

7 Priority Schools are schools with characteristics that correlate with a low propensity to respond and have an impact on estimates.

8 The Welcome Text will be sent at the time of Text 1.

9 Good, K. (1997). A study of factors affecting responses in electronic mail surveys. Dissertation, Western Michigan University, DAI, vol 58-10A pg. 3899, 119 pages

10 Carroll, S.H. & Zukerberg, A. (2013, November 4-6). Do Names Matter? Experiments Comparing Different Branding and Levels of Personally Identifiable Information in a Mail Questionnaire [Proceedings of a research conference]. Federal Committee on Statistical Methodology (FCSM), Washington, DC, U.S.

11 Dillman, D. A., Lesser, V., Mason, R., Carlson, J., Willits, F., Robertson, R., et al. (2007). Personalization of mail surveys for general public and populations with a group identity: Results from nine studies. Rural Sociology, 72, 632–646. doi:10.1526/003601107782638693.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	tfs omb supporting statement [Memorandum]
Author	NCES
File Modified	0000-00-00
File Created	2022-03-23