Justification

Volume I:

Fast Response Survey System (FRSS) 108: Career and Technical Education (CTE) Programs in Public School Districts – Pretest

OMB# 1850-0803 v.166

August 2016

National Center for Education Statistics

U.S. Department of Education

Justification

The National Center for Education Statistics (NCES), U.S. Department of Education (ED), requests OMB approval under the NCES system clearance for Cognitive, Pilot and Field Test studies (OMB #1850-0803) to conduct pretest calls for Fast Response Survey System (FRSS) survey #108 on career and technical education (CTE) programs in public school districts. The Office of Career, Technical, and Adult Education (OCTAE) requested that NCES conduct this FRSS survey.

Nearly all public high school students (95 percent of ninth-grade students in 2009) attended a school that offered Career and Technical Education (CTE) instruction, either on campus or at a partnering school. In 2009, 85 percent of public high school graduates had completed one or more occupational CTE courses, 76 percent had earned at least one full credit in occupational CTE, and 19 percent were CTE concentrators who had earned at least three credits in the same CTE field.¹

Effective, high-quality CTE programs are aligned with college- and career-readiness standards as well as the needs of employers, industry, and labor. They provide students with a curriculum that combines integrated academic and technical content and strong employability skills. They also provide work-based learning opportunities that enable students to connect what they are learning to real-life career scenarios and choices. The students participating in effective CTE programs graduate with industry certifications or licenses and postsecondary certificates or degrees that prepare them for in-demand careers within high-growth industry sectors.²

The purpose of the survey pretest is to identify and correct any potential issues with the content and format of the survey before conducting full scale implementation, to ensure that the survey captures the intended meaning of each question and minimizes the burden imposed on respondents. The request to conduct the full-scale survey will be submitted at a later date under OMB generic clearance for quick response surveys (OMB#1850-0733), which are authorized under the Education Science Reform Act of 2002 (ESRA, 20 U.S.C. §9543). NCES has contracted Westat for all stages of this survey.

Design

Overview of Survey Development

NCES has contracted Westat to administer the FRSS 108survey, including development of the survey instrument. FRSS has established procedures for developing short surveys on a wide variety of topics. The techniques used to shape the survey design on FRSS 108 include literature reviews on CTE programs, input from the NCES Quality Review Board (QRB), several rounds of feasibility calls, and up to two pretests. The specific ways we plan to use pretest calls are discussed below.

The current survey reflects lessons learned from topics and issues identified through literature review, with modifications based on three rounds of feasibility calls with public school district personnel most knowledgeable about high school CTE programs. The first round of feasibility calls was conducted with 8 respondents in October and November 2015 (OMB# 1850-0803 v. 144). Because this is a new survey topic, the first round of calls used an open-ended interview guide to learn more about the CTE programs that districts offer to high school students, the terminology districts use regarding these programs, and the characteristics of the CTE programs they offer. The second round of feasibility calls was conducted with 11 respondents in January and February 2016 during which respondents provided feedback on draft survey questions. The third round of feasibility calls was conducted with 15 respondents in June and July 2016 during which respondents were asked to review draft survey questions, instructions, and definitions based on the initial rounds of feasibility calls. During the pretest, we will test all questions on the questionnaire and also obtain estimates of the respondent time required to complete the survey.

NCES Review and Consultations Outside of Agency

The NCES QRB members reviewed a draft list of questionnaire topics prior to the submission of the OMB package for the feasibility calls. Revisions were made to the list of topics based on input from the reviewers, and the list was used to develop an interview guide for the feasibility calls. After the second round of feasibility calls, a draft questionnaire was developed with input from OCTAE. During the third round of the feasibility calls, revisions were made to the draft questionnaire with input from OCTAE. Following the last round of feasibility calls, the QRB reviewed the draft questionnaire, and revisions were made based on their input. The revised version will be used for the pretest. In addition to staff from each of the three Divisions at NCES, the QRB also included staff from OCTAE and the Office of Planning, Evaluation, and Policy Development (OPEPD). The QRB members for this survey are listed below:

Braden Goetz, OCTAE

Sharon Lee Miller, OCTAE

Lul Tesfai, OCTAE

Kelly Fitzpatrick, OPEPD

Michael Fong, OPEPD (Policy and Program Studies Service)

Milagros Lanauze, OPEPD (Budget Service)

John Ralph, NCES (Annual Reports and Information)

Jing Chen, NCES (Assessment Division)

Heidi Silver-Pacuilla, NCES (Assessment Division)

Chris Chapman, NCES (Sample Surveys Division)

Sharon Boivin, NCES (Sample Surveys Division)

Lisa Hudson, NCES (Sample Surveys Division)

Elise Christopher, NCES (Sample Surveys Division)

Gigi Jones, NCES (Administrative Data Division)

Joseph Murphy, NCES (Administrative Data Division)

Kashka Kubzdela, NCES (Statistical Standards and Data Confidentiality)

Sample, Burden, and Cost

In this submission, we are requesting approval to conduct up to two rounds of pretest calls with the revised questionnaire, with 15 or fewer respondents in each round from school districts around the nation. School districts will be recruited to participate in pretest calls based on various district characteristics including level (secondary or unified), size, urbanicity (locale), and geographic region. Respondents will be recruited by telephone and will be identified as the person in the district who is most familiar with CTE programs for high school students in their district.

Telephone interviewers will recruit participants for the pretest calls using the recruitment script in Attachment 1. Interviewers will schedule an appointment to complete the pretest calls with cooperating district-level personnel. Following telephone recruitment, interviewers will email a cover letter and draft questionnaire to the participating districts (as discussed below in the Data Collection Instrument section). Respondents will be asked to review, complete, and fax back the paper and pencil questionnaire, and will be scheduled to participate in a telephone debriefing.

In order to recruit 15 respondents per round, we anticipate contacting 45 public school districts per round (Table 1). On average, recruitment calls with respondents who agree to participate in the pretest calls are expected to take about 10 minutes to explain the purpose of the pretest and set up an appointment to discuss the questionnaire. All other recruitment calls are expected to take about 3 minutes. The questionnaire and the pretest debriefing interview are each expected to take respondents approximately 30 minutes to complete. The total estimated burden is approximately 40 hours for two rounds of pretest calls. We anticipate that the estimated cost to the federal government will be approximately $5,000 for each round of pretest calls.

Table 1. Maximum burden time for each of up to two rounds of pretest calls for FRSS 108

Respondents	Number of Respondents	Number of Responses1	Burden Hours per Respondent	Total Burden Hours
Recruitment – Districts not participating in the pretest	30	30	0.05	2
Recruitment – Districts participating in the pretest	15	15	0.17	3
Pretest questionnaire debrief	15	15	1.00	15
Total per round	45	60	-	20
Total for two rounds	90	120	-	40

¹ Counts each contact (e.g., recruitment and debriefing interview with the same respondents are counted separately).

Data Collection Instrument

For each round of pretest, a cover letter and draft questionnaire will be emailed to each participating school district. The cover letter and questionnaire appear in attachments 2 and 3. The cover letter thanks the respondent for agreeing to participate in the pretest, introduces the purpose and content of the survey, indicates that participation is voluntary, indicates that respondents should complete the questionnaire and fax it back to Westat, includes questions for respondents to consider while completing the questionnaire, and provides contact information should any questions arise before the scheduled discussion with the survey manager. On the cover letter and on the cover of the survey, respondents are assured that their participation is voluntary and their answers may not be disclosed or used in identifiable form for any other purpose unless compelled by law. The public law is cited on the cover letter and the front page of the survey. The materials for the second pretest round (if it takes place) will be similar, except the survey instrument will include the modifications that will result from the first round. The current instrument is discussed below.

Questionnaire

The purpose of this Fast Response Survey System (FRSS) survey is to collect nationally representative data from public school districts about CTE programs offered by the districts. These programs may be offered at district facilities or in a partnering off-site location, such as area CTE facilities or postsecondary institutions. The sample will focus on school districts with high schools because CTE programs are generally designed for high school students.

Questions in the survey are based on criteria identified by OCTAE in their blueprint specifications for the most recent reauthorization of the Perkins Act as characteristics that define high-quality CTE programs. The questionnaire covers topics that focus on the characteristics of the CTE programs offered by districts, work-based learning activities, involvement of employers in CTE programs, barriers to offering programs and to participation in CTE, and the factors districts consider for adding or phasing out CTE programs.

The instructions and definitions page provides the definition of career and technical education (CTE) programs as “a sequence of courses at the high school level that provides students with the academic and technical knowledge and skills needed to prepare for further education and careers in current or emerging professions.” On this page, respondents are instructed to include all CTE programs that the district offers to high school students, including programs provided by the district or by other entities; to report only for CTE programs offered to high school students; and to report for the 2016–17 school year and the summer of 2016.

The box above question 1 provides the following instruction to respondents: “For this survey, include all CTE programs that your district offers to high school students, including programs provided by your district or by other entities (such as an area/regional CTE center, a consortium of districts, or a community or technical college).”

Question 1 is a yes/no question that asks whether the district offers CTE programs to students at the high school level. Respondents who answer “no” are directed to skip to question 16 because the intervening questions only apply to districts that offer CTE programs to their high school students.

Question 2 asks respondents to indicate (yes or no) whether each of the listed entities provide any of the CTE programs that the district offers to their high school students. The entities include the following: an area/regional CTE center or a group/consortium of school districts, your district individually (not as part of a consortium), 2-year community or technical college(s), and 4-year college(s) or universities. Respondents can also specify “other.”

Question 3 asks respondents to indicate (yes or no) whether the district offers CTE programs to high school students at each of the listed locations. The locations include the following: at some or all of your district’s regular (comprehensive) high schools; at another district’s regular (comprehensive) high school; at a CTE-focused high school that students attend full time; at a CTE center that students attend part time (for example, students spend half the day at the CTE center and half at the regular high school); and at a 2-year community or technical college, or a 4-year college or university. Respondents can also specify “other.”

Question 4 asks the respondent to report the number of CTE programs the district offers to high school students.

Question 5 asks about how many of the CTE programs offered by the district to high school students are structured as career pathways that align with related postsecondary programs. The response options are none, few, some, most, and all.

Question 6 is a yes/no question that asks whether the district offers any CTE courses in which students may earn high school credits in math, science, English/language arts, or social studies.

Question 7 is a yes/no question that asks whether the district offers any CTE courses for which students can earn both high school and postsecondary credits for the same course.

Question 8 is a yes/no question that asks whether the district offers any CTE courses online, including courses in a blended/hybrid format.

Question 9 asks respondents to indicate (yes or no) whether each of the listed items is included in any of the CTE programs offered by the district to high school students. The items include the following: student-run enterprises or services (for example, school store or restaurant, cosmetology services, automotive or construction services, child development facility); mentoring by local employers; on-the-job training, internships, practicums, clinical experiences, or cooperative education (co-op); and apprenticeships or pre-apprenticeship programs (such as youth apprenticeships). Respondents can also specify “other work-based learning.”

Question 10 asks about how many of the CTE programs offered by the district to high school students require work-based learning activities (such as those listed in Question 9) for completion of the program. The response options are none, few, some, most, or all.

Question 11 asks respondents to indicate the extent to which employers are involved with the CTE programs offered by the district to high school students in each of the listed ways. The listed ways include the following: provide work-based learning opportunities, serve on your district’s CTE advisory council, advise about which occupations are in demand, provide advice on CTE programs to add or eliminate, review CTE program curriculum, provide guidance on industry standards, provide guidance about equipment or facilities, donate equipment, host student field trips, serve as guest speakers to CTE students, provide guidance for student CTE projects, judge student CTE competitions, and provide training opportunities for CTE teachers. Respondents can also specify “other.” The response options are not at all, small extent, moderate extent, large extent, and very large extent.

Question 12 asks respondents to indicate how much of a barrier each of the listed items is to the district in offering CTE programs to high school students. The items include the following: lack of funding or high cost of programs (for example, cost of infrastructure or equipment); facilities or space limitations; finding or keeping teachers for in-demand industries and occupations; limited availability of professional development in technical fields; difficulty keeping CTE teachers’ technical skills up to date; CTE teachers who move into teaching from other occupations have difficulty obtaining a regular or standard state teaching certificate; and difficulty developing partnerships with employers for work-based learning. Respondents can also specify “other.” The response options are not a barrier, small barrier, moderate barrier, large barrier, and very large barrier.

Question 13 asks respondents to indicate how much of a barrier each of the listed items is to student participation in the CTE programs offered by the district to high school students. The items include the following: lack of time in students’ schedules for CTE courses; students’ or parents’ negative perceptions of CTE; transportation to CTE programs outside of the high school campus; transportation for work-based learning; students’ costs for supplies, uniforms, or materials; students’ difficulty finding work-based learning opportunities; lack of student support services for special populations. Respondents can also specify “other.” The response options are not a barrier, small barrier, moderate barrier, large barrier, and very large barrier.

The box above question 14 provides the following instruction to respondents: “Questions 14 and 15 ask about adding or phasing out CTE programs. Please answer these questions about CTE programs for which your district has a role in making these decisions. Check here and skip to question 16 if your district does not have a decision-making role in adding or phasing out CTE programs.” This instruction and checkbox is included because during feasibility calls, some districts reported that they were not involved in the decisions for adding or phasing out CTE programs that were provided by entities other than the district, such as a community college or an area/regional CTE center.

Question 14 asks respondents to indicate the extent to which each of the listed factors influences the district’s decision on whether to add a new CTE program for high school students. The factors include the following: student interest; facilities/space considerations (for example, whether appropriate space is available); costs for new program; availability of qualified teachers; information on which industries and occupations are in demand; employer (business/industry) recommendations; postsecondary institution recommendations; recommendations from your state department of education; career pathways from the high school to the postsecondary level (for example, to structure new pathways or better align existing pathways). Respondents can also specify “other.” The response options are not at all, small extent, moderate extent, large extent, and very large extent.

Question 15 asks respondents to indicate the extent to which each of the listed factors influences the district’s decision on whether to phase out a CTE program for high school students. The factors include the following: enrollment or student interest; facilities/space considerations (for example, facilities are outdated, space is needed for other purposes); cost of program; availability of qualified teachers (for example, a teacher leaves and is difficult to replace); information on which industries and occupations are in demand; employer (business/industry) recommendations; postsecondary institution recommendations; recommendations from your state department of education; career pathways from the high school to the postsecondary level (for example, if a program does not align with a career pathway). Respondents can also specify “other.” The response options are not at all, small extent, moderate extent, large extent, and very large extent.

Question 16 asks whether high school students within the responding district’s enrollment area have the option of enrolling in a separate CTE district instead of enrolling in the responding district. Because some respondents may not be familiar with this situation, the question first explains that some states have CTE school districts that provide only CTE programs and students have the option of enrolling in the CTE district instead of enrolling in their home district. All districts (both those that offer CTE programs and those that do not) are asked to answer Q16. For districts that do not offer CTE programs to their enrolled students (question 1 = no), it is especially important to know whether those students had the option of enrolling in a CTE district.

Timeline

Pretest activities are expected to begin as soon as approval is received from OMB. It is anticipated that recruitment, debriefing, write up of the memorandum summarizing the results, and survey revisions will take approximately 4-6 weeks for each pretest round.

Attachment 1: FRSS 108 Pretest Call Recruitment Script

FRSS 108

Career and Technical Education Programs in Public School Districts

Pretest Call Recruitment Script

Hello, my name is __________________.

I am calling from Westat on behalf of the National Center for Education Statistics, within the U.S. Department of Education, regarding a survey on career and technical education (CTE) programs in public school districts. I would like your help in identifying the district-level person who is most familiar with CTE programs in your district.

Who at your district is most knowledgeable about career and technical education programs for high school students? (This is often a director or coordinator of career and technical education programs at the district level.)

(RECORD CONTACT INFORMATION)

May I please speak to that person?

CONNECTED TO DISTRICT–LEVEL PERSON MOST FAMILIAR WITH CAREER AND TECHNICAL EDUCATION PROGRAMS FOR HIGH SCHOOL STUDENTS

Hello, my name is __________________.

I’m calling from Westat on behalf of the National Center for Education Statistics, within the U.S. Department of Education, regarding a survey on career and technical education (CTE) programs. We would like your help in reviewing our draft questionnaire to ensure that it is clear and easy to complete. Specifically, we would like you to complete a questionnaire and fax it back to us, and then obtain your comments about the survey by telephone. This is a short questionnaire which should take about 30 minutes to complete. [The call will take about 30 minutes.]

Your input, while voluntary, will be essential in developing a questionnaire that is relevant. All information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose unless otherwise compelled by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573).

1. May I have your email address to send you the survey materials?

2. We ask that you complete the questionnaire and fax it back to us before you talk to the survey manager. When would be a good time for the survey manager, Cindy Gray, to call you to discuss the survey and obtain your comments? How about [SUGGEST A TIME]. [Just to be sure, you are in the [Eastern, Central, Mountain, Pacific] time zone?]

3. What is the best telephone number at which the survey manager can reach you?

Thank you. Your insights will be very helpful

Attachment 2: FRSS 108 Pretest Cover Letter

U.S. Department of Education  Institute of Education Sciences  National Center for Education Statistics

[Date] 2016

Dear Pretest Participant,

Thank you for agreeing to participate in the pretest of the survey on career and technical education programs in public school districts. Westat, a research company located in Rockville, Maryland, is administering this survey for the National Center for Education Statistics (NCES), within the U. S. Department of Education. The survey is part of the Fast Response Survey System (FRSS), a system charged with collecting information on important and emerging issues related to education. The survey is designed to be completed by district personnel most knowledgeable about career and technical education (CTE) programs for high school students. The purpose of the survey is to provide nationally representative data about CTE programs in U.S. public school districts.

We are currently conducting the pretest of the survey. Your input, while voluntary, will be essential in developing a survey that is relevant, clear, and not overly burdensome to respondents. Your participation is very important because your comments will improve the survey before the actual data collection begins. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. 9573.)

We ask that you (1) complete the questionnaire and fax it to Westat; (2) keep track of the time you spend filling out the questionnaire; (3) write down any comments about the questionnaire; and (4) discuss your comments with me by telephone at the time scheduled. Please keep the following questions in mind as you complete the questionnaire:

1. Are the instructions and definitions clear and helpful to you?

2. Are the survey questions clear and easy to interpret?

3. Would you have access to the information necessary for answering these questions?

4. How long did it take you to answer the survey?

Please fax the completed questionnaire to me. My toll-free fax number is 1-800-254-0984. My colleague and I will call you at the scheduled time to get your feedback on the questionnaire and discuss any comments or suggestion you may have about the study. In the meantime, feel free to call me at Westat’s toll-free number,
800-937-8281, ext. 4336, if you have any questions. You may also reach me by email at cindygray@westat.com.

Thank you for your much needed assistance!

Sincerely,

Cindy Gray

Westat Survey Manager

1 U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service (2014), National Assessment of Career and Technical Education: Final Report to Congress. Washington, DC.

2 U.S. Department of Education, Office of Career, Technical, and Adult Education, Investing in America’s Future: A Blueprint for Transforming Career and Technical Education (Summary) (2012). Washington, DC.

Conducted by WESTAT  1600 Research Blvd.  Rockville, MD 20850  (301) 251-1500  TOLL FREE (800) 937-8281