National Assessment of Educational Progress (NAEP) 2017 Feasibility Study of Middle School Transcript Study (MSTS)

Customized TUDA-level data summaries at no cost to TUDAs. Each participating TUDA will receive a data summary containing analyses specific to issues important to their district, deepening their understanding of topics such as students’ course trajectories, curriculum rigor, and achievement gap problems. MSTS staff will work with each TUDA to customize these data summaries at no cost to the districts. Note that given this is a feasibility study, the summaries will be unofficial and cannot be published or shared outside of the district office.

Expanded TUDA-level NAEP analysis with the MSTS feasibility study data. Data collected from the MSTS feasibility study will provide TUDAs more contextual data to explain their eighth-grade students’ NAEP assessment scores. What courses they took and their performance in those courses, combined with their attitudes toward the courses as asked in the student questionnaire, will provide a more complete picture of how students perform on the eighth-grade NAEP assessments.

New measures to focus on key issues in middle school at TUDA-level. Measures of middle school curriculum levels will be developed to look at the rigor of middle school curriculum and examine academic tracking issues. The MSTS feasibility study will also be able to take a comprehensive look at the digital divide issue at the middle school level by combining data from the NAEP questionnaires and the information about online courses provided by the TUDAs. By examining student coursetaking patterns, maladaptive coursetaking behaviors, and other factors that are typically associated with student dropouts (i.e. as absenteeism, and grade level/course repeats), middle school success indicators could be developed to further identify students who are at risk of dropping out.

No burden to schools and students. Unlike previous transcript studies where data collections are done at the school level, data collection for the Middle School Transcript Study (MSTS) 2017 feasibility study will be done at the district level, thereby placing no burden on schools and students.

Minimal labor cost to the districts for data collection. Because the MSTS 2017 feasibility study will be a part of the NAEP 2017 Grade 8 assessments, there will be no cost to the districts, except for minimal labor cost from TUDA data personnel to extract and prepare electronic data files to submit for this study.

Appendix B: List of all 27 TUDA districts

NAEP TUDA District	Year Joined
Albuquerque Public Schools	2011
Atlanta Public Schools	2002
Austin Independent School District	2005
Baltimore City Public Schools	2009
Boston Public Schools	2003
Charlotte-Mecklenburg Schools	2003
Chicago Public Schools	2002
Clark County School District (Las Vegas, NV)	2017
Cleveland Metropolitan School District	2003
Dallas Independent School District	2011
Denver Public Schools	2017
Detroit Public Schools	2009
District of Columbia Public Schools	2002
Duval County Public Schools (Jacksonville, FL)	2015
Fort Worth Independent School District (Fort Worth, TX)	2017
Fresno Unified School District	2009
Guilford County Schools (Greensboro, NC)	2017
Hillsborough County (FL) Public Schools	2011
Houston Independent School District	2002
Jefferson County Public Schools (Louisville, KY)	2009
Los Angeles Unified School District	2002
Miami-Dade County Public Schools	2009
Milwaukee Public Schools	2009
New York City Public Schools	2002
San Diego Unified School District	2003
School District of Philadelphia	2009
Shelby County (Memphis, TN)	2017

Appendix C: NAEP 2017 MSTS Feasibility Study Research and Development (R&D) Report and TUDA Individual Data Summaries Analysis List

NAEP 2017 MSTS Feasibility Study Main Research and Development (R&D) Report Analysis List

The following list includes analyses that are expected to be included in the NAEP 2017 MSTS Feasibility Study R&D report. Because the study focuses on the TUDAs in the NAEP sample, the analyses will be presented at the TUDA level. There will be no national-level analyses presented in the report.

Carnegie credits earned by students
- Focus on combined count (6^th-8^th grades) instead of individual grade level count
- Breakout by student gender and race/ethnicity
- Breakout by academic subjects (English/language arts, math, science, social studies)
- May include foreign languages, arts, or introductory CTE as additional subjects

Grade point average earned by students
- Focus on individual grade levels (6^th, 7^th, 8^th grades) instead of a single combined GPA
- Breakout by student gender and race/ethnicity
- Breakout by academic subjects (English/language arts, math, science, social studies)

Curriculum rigor
- Three levels for the measure: Basic, Standard, and Rigorous
- Rigorous covers honor courses, Algebra I, Geometry, Biology, and first year of high school foreign language
- Standard covers appropriate grade-level courses, Pre-Algebra, Life and Physical Sciences, and middle school foreign language courses
- Basic covers developmental courses, Basic/General Math, and General Science courses
- Report percentages of students attaining each level
- Breakout by student gender and race/ethnicity

Advanced coursetaking
- Percentages of students taking Algebra I, Geometry, Biology, and high-school-level foreign language courses
- Breakout by student gender and race/ethnicity

NAEP reading and mathematics assessment scores
- Compare to curriculum rigor categories
- Compare to advanced coursetaking (yes/no if they took honors English/language arts, Algebra I, Geometry)
- Breakout by student gender and race/ethnicity

Potential dropouts
- Look at the coursetaking measures (credits earned, curriculum rigor, etc.) and other characteristics (age, absent days, grade level/course repeats, etc.) to identify potential dropout candidates within the groups
- Will likely involve derived variables and/or indices involving a combination of available data from student transcripts
- Will likely develop a national definition of potential dropout students to use for the NCES R&D report and apply it across all TUDAs

Digital divide
- Involves NAEP Grade 8 student questionnaire items directly related to computer and Internet usage
  - Access to Internet (Question 9a)
  - Desktop, laptop, tablet, smartphone access (Questions 9e, 9f, 9g)
  - How often use Internet for homework (Question 10)
  - How often used desktop, laptop, tablet in classes (Questions 18 and 19)
  - How much time using computer to work on English/mathematics schoolwork and homework (Question 29)
  - Computer-related events during math class (Question 32)
  - Computer-related questions about using computer for math homework (Questions 33, 34, and 35)
  - Activities outside of school that indicate access to computer and/or internet (Question 37)
- Can look at variables individually or combine them into an index value (or use NAEP’s index value if available)
- Compare against academic tracking/curriculum rigor and advanced coursetaking
- Compare NAEP scores through crosstabs of digital-related/technology variables and selected coursetaking measures

NAEP 2017 MSTS Feasibility Study TUDA Individual Data Summary Analysis List

The following list includes potential analyses that may be included in the NAEP 2017 MSTS Feasibility Study individual data summaries prepared for each participating TUDA. Each data summary will only focus on an individual TUDA and not include data from the other participating TUDAs. Some analyses are repeated from the NAEP 2017 MSTS Feasibility Study R&D report, but may have additional detail specific to the individual TUDA.

Carnegie credits earned by students
- Focus on combined count (6^th-8^th grades) instead of individual grade level count
- Breakout by student gender and race/ethnicity
- Breakout by academic subjects (English/language arts, math, science, social studies)
- May include foreign languages, arts, or introductory CTE as additional subjects, based on the individual TUDA’s eighth graduation requirements

Grade point average earned by students
- Focus on individual grade levels (6^th, 7^th, 8^th grades) instead of a single combined GPA
- Breakout by student gender and race/ethnicity
- Breakout by academic subjects (English/language arts, math, science, social studies)
- May include foreign languages, arts, or introductory CTE as additional subjects, based on the individual TUDA’s eighth graduation requirements

Academic coursetaking tracks
- Similar to curriculum rigor, but defined based on the courses that the TUDA offered to their students
- Three levels for the measure: Essential, Regular, and Advanced
- Advanced covers high-level courses available to the students in the TUDAS (such as Algebra I and honors courses)
- Regular covers the courses the students typically take (such as Pre-Algebra and grade-appropriate English/language arts courses) in middle school
- Essential covers the courses offered to students who are struggling academically (such as Basic/General Math and developmental courses)
- Report percentages of students in each track
- Breakout by student gender and race/ethnicity

NAEP reading and mathematics assessment scores
- Compare to academic coursetaking tracks
- Compare to the eighth-grade English/language arts and mathematics courses the students took
- Comparisons include breakout by student gender and race/ethnicity

State and district assessment scores
- Compare state and/or district assessment scores to similar coursetaking measures as the NAEP assessment scores
  - Academic coursetaking tracks
  - Eighth-grade courses taken by students that match the subjects of the assessments
- Breakout by student gender and race/ethnicity for all comparisons
- Explicitly depends on TUDA’s interest and willingness to share relevant state/district assessment scores with Westat analysts

Potential student dropouts
- Focus on the various methods of defining dropout risks that the individual TUDA uses
- Incorporate TUDA-specific data collected during the MSTS feasibility study
  - How many days students are absent
  - Whether a student was held back within sixth, seventh, or eighth grades
  - Whether a student repeated a core subject course
    - Total number of courses repeated during middle school
    - Breakout by subject areas
    - Breakout by grade level
  - English/language arts, mathematics, science, and social studies courses taken by students and the grades they earned

Mathematics and science curriculum sequences
- Identify common mathematics and science coursetaking sequences from sixth through eighth grades
- Identify students whose transcripts identify that they need require additional resources to complete their mathematics and science courses (such as taking resource-based classes or additional tutoring sessions)
- Compare mathematics coursetaking sequences with NAEP eighth-grade mathematics assessment scores
- Comparisons include breakout by student gender and race/ethnicity (if the student sample size allows such comparisons)

In addition, when discussing the study with the TUDAs, there may be potential analyses brought up during the conversations that could be included in the individual TUDA data summaries if (a) there was enough interest from the participating TUDAS, and (b) the analyses are possible with the student transcript information that is being collected.

Appendix D: Sample Recruitment Letter from NCES

Dear TUDA District Superintendent________(each letter to be individually addressed):

I am writing to invite your district to participate in a new initiative that will expand the National Assessment of Educational Progress (NAEP) to include a transcript study focusing on middle school grades. In 2017, the National Center for Education Statistics (NCES) will conduct a feasibility study of the collection of middle school student coursetaking information for students selected for the NAEP grade 8 assessments. This feasibility study will rely solely on electronic submission of course catalogs and student transcripts and will provide educators and policymakers a new resource for exploring the relationship between students’ coursetaking patterns and their performance on NAEP.

The feasibility study of the groundbreaking Middle School Transcript Study (MSTS) is the first NAEP transcript study to focus on middle school grades. While it may eventually be expanded to schools across the country, your district and other large urban districts are the first to have an opportunity to participate. This is also the first transcript study to rely entirely on electronic submissions, which will substantially reduce cost, time, and effort over previous transcript studies. In return, you will receive detailed district-level analyses.

Costs for this feasibility study will be the responsibility of the federal government, as with NAEP assessments at grades 4 and 8. NCES contractors will work with your district staff to obtain electronic course catalogs and student transcripts; this is the only additional burden placed on your district. There is no additional burden at the school level. The MSTS is part of an overall movement in NAEP transcript studies toward relying on more efficient electronic records.

To meet critical deadlines for data collection preparation, it is necessary to identify interested districts now. In the event that NCES receives letters of interest from more districts than funding can accommodate, we will select a sample of districts that is as regionally representative and demographically diverse as possible, while also considering the quality of electronic records in each district.

NCES will host a WebEx on [DATE] for jurisdictions interested in participating in the study to review the details of the study, including the study timeline, data to be collected as part of the transcripts, and reporting plans. Details about the WebEx will be emailed to your NAEP TUDA Coordinator.

If your district wishes to participate in this innovative feasibility study, please complete and submit the enclosed form by close of business on [DATE]. Districts will be notified of their participation by [DATE].

Thank you for considering participation in this important activity. NCES will keep you apprised of the status of the feasibility study. If you have any questions, please contact Linda Hamilton at 202-245-6360 or linda.hamilton@ed.gov.

Sincerely,

Peggy G. Carr, Ph.D.

Acting Commissioner

National Center for Education Statistics

cc: Mike Casserly, Council of the Great City Schools (CGCS)

TUDA Coordinator (edit with TUDA coordinator name)

Enclosures: Benefits to TUDAs [Appendix A], Brochure [Appendix E], Letter of Interest [Appendix F]

Appendix E: TUDA Recruitment Brochure Text

COVER

NAEP 2017 Middle School Transcript Study (MSTS) Feasibility Study

INTERIOR

What is the Feasibility Study of the NAEP 2017 Middle School Transcript Study?

As part of the National Assessment of Educational Progress (NAEP) 2017 grade 8 assessments, the feasibility study of the NAEP 2017 Middle School Transcript Study (MSTS) is the first landmark NAEP study to examine the feasibility of linking middle school students’ coursework from sixth through eighth grades with NAEP 2017 grade 8 assessment data.

The NAEP 2017 MSTS feasibility study is also the first NAEP transcript study to provide student coursetaking data at the district level and to rely solely on the electronic submission of course catalogs and student transcripts.

The NAEP 2017 MSTS feasibility study is conducted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education. The data collection will be a part of the NAEP 2017 assessments for mathematics, reading, and writing.

Why is NAEP 2017 MSTS feasibility study important?

The NAEP 2017 MSTS feasibility study provides valuable student coursetaking data that supplements NAEP assessment data along with school, teacher, and student questionnaire responses. It allows for the comparison of student coursetaking data within jurisdictions and informs the decision if additional Middle School Transcript Studies should be conducted in the future. In addition, the MSTS feasibility study relies on electronic school records for collecting data for this and future transcript studies, which eliminates the burden on individual schools to provide paper copies of transcripts.

Why participate in NAEP 2017 MSTS feasibility study?

The study promotes a deeper understanding of student academic achievement by examining coursetaking patterns, middle school curriculum rigor, and potential student academic tracking. Coupled with the NAEP school, teacher, and student survey questionnaire information, this study provides valuable information to help identify academic coursetaking behaviors and possible academic pathways to success.

Study Timeline:

Winter 2017 Participation Recruitment

Spring/Summer 2017 Data Collection

Spring/Summer 2018 Reporting

Requirements for Participation in the NAEP 2017 MSTS Feasibility Study

Districts interested in participating in the NAEP 2017 MSTS feasibility study must have the following:

An electronic school records system and the ability to transmit student records electronically,
The ability to provide electronic course catalogs containing courses available to students from grades 6 through 8, and
The course ID number available on the electronic student record serving as a linking variable to match course numbers in the course catalogs.

BACK

The National Assessment of Educational Progress (NAEP) is a congressionally mandated project conducted by the National Center for Education Statistics (NCES), within the U.S. Department of Education and the Institute of Education Sciences.

NCES is the primary federal entity for collecting, analyzing, and reporting data related to education in the United States. It fulfills a congressional mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in the United States; conduct and publish reports and specialized analyses of the meaning and significance of such statistics; and assist state and local education agencies in improving their statistical systems.

NCES activities are designed to address high-priority education data needs; provide consistent, reliable, complete, and accurate indicators of education status and trends; and report timely, useful, and high-quality data to the U.S. Department of Education, the Congress, the states, other education policymakers, practitioners, data users, and the general public.

For questions about the NAEP 2017 MSTS feasibility study, please email MSTS@westat.com.

Appendix F: Letter of interest from districts

[Insert Date]

Linda Hamilton

National Center for Education Statistics

Potomac Center Plaza

550 12th Street, SW

Room 4093

Washington, D.C. 20202, USA

Dear Ms. Hamilton

_____________________________________ (insert TUDA District’s name) is interested in participating in the feasibility study of the NAEP 2017 Middle School Transcript Study (MSTS). I understand that, if NCES receives letters of interest from more TUDA districts than funding can accommodate, they will select a sample of districts that is as regionally representative and as demographically diverse as is possible; the quality of electronic school records may also be taken into consideration. Thank you very much.

Sincerely,

(signature)

Please mail to:

Linda Hamilton

National Center for Education Statistics

Potomac Center Plaza

550 12th Street, SW

Room 4093

Washington, D.C. 20202, USA

Appendix G: Letter to districts selected for the study

[Date]

[Supt. Name]

[Supt. Address]

Dear Superintendent [Last name]:

Thank you for your interest in participating in the NAEP 2017 Middle School Transcript Study (MSTS) feasibility study. I am pleased to inform you that your TUDA district has been selected for the feasibility study. A list of all TUDA districts selected to participate is included with this letter.

As I noted in my [date interest letter sent] letter to you, administration costs for this study will be the responsibility of the federal government. Procedures for administering the study will involve district staff assisting with data collection, including the extraction, preparation, and submission of electronic data files. NAEP contractors will work to minimize the administrative burden on your district.

Staff from Westat, the NAEP sampling and data collection contractor, will contact your NAEP TUDA Coordinator to discuss the collection of district-level course catalogs and student transcripts. Since the MSTS feasibility study data collection is covered as part of the NAEP data sharing agreement, no additional data sharing agreements should be needed.

My colleagues and I at NCES are looking forward to working with you on this new aspect of the NAEP assessment program.

Sincerely,

Peggy G. Carr, Ph.D.

Acting Commissioner

National Center for Education Statistics

Cc: TUDA Coordinator

Appendix H: Letter to districts not selected for the study

[Date]

[Supt. Name]
[State Doe Name]
[Address]
[City, State Zip]

Dear TUDA District Superintendent [Last Name]:

Thank you for your interest in participating in the NAEP 2017 Middle School Transcript Study (MSTS) feasibility study. Because we received letters of interest from more NAEP TUDA districts than funding can accommodate, we were tasked with selecting a sample of districts that is as regionally representative and demographically diverse as possible. The capability and quality of electronic student records were also part of the consideration for study inclusion. Based on these criteria, we were not able to select your district for participation in this study.

We appreciate your interest in participating in this study. Although unlikely, in case a selected district can no longer participate, we may ask you if your district still wishes to participate.

Thank you again for your time. We look forward to possibly working with you for the next iteration of this study.

Sincerely,

Peggy G. Carr, Ph.D.

Acting Commissioner

National Center for Education Statistics

Appendix I: Requested Data for the Feasibility Study of the Middle School Transcript Study

TUDA educational information

Eighth-grade graduation requirements

Credits earned for completing a year-long course

Common grading standard (for course grade standardization)

State or district assessments students take in grades 6-8 (if any)

District-wide educational programs available to students in grades 6-8

TUDA district catalog

Course name

Course ID number (linkable to transcripts)

Course description (if available)

Credits earned for passing the course

Special education course designation

English Language Learner course designation

Online course designation

Student ID and demographic information

Unique Student Identifier (to match with NAEP)

NAEP assessment student took (will get from NAEP)

Month of birth (will get from NAEP)

Year of birth (will get from NAEP)

Gender (will get from NAEP)

Race/ethnicity (will get from NAEP)

NSLP status (will get from NAEP)

IEP/ELL status (will get from NAEP)

Student educational information

Eighth-grade graduation status (yes/no)

Number of days absent (sixth, seventh, eighth grade)

Student transfer into TUDA between grades 6-8? (yes/no)

Number of credits earned (sixth, seventh, eighth grade)

Grade point average (sixth, seventh, eighth grade)

State and/or district assessment scores (if requested)

Transcript record data

Course name

Course ID number (linkable to catalog)

Grade level taken

School year taken

Length of course (semester, quarter, etc.)

Number of credits earned

Grade earned (letter or numeric)

Level of course (regular, honors, etc.)

Location, if not taught at school

Special education course designation

English Language Learner course designation

Transfer course designation

Appendix J: Data Sharing Agreement Template

U NITED STATES DEPARTMENT OF EDUCATION

INSTITUTE OF EDUCATION SCIENCES

NATIONAL CENTER FOR EDUCATION STATISTICS

Data Sharing Agreement

2017 National Assessment of Educational Progress (NAEP)

This document establishes a data security agreement between ___________________ and the National Center for Education Statistics of the U.S. Department of Education for the 2017 NAEP assessment program.

NAEP is a congressionally mandated project of the U.S. Department of Education. P.L 107-279, Title III, directs the Commissioner for Education Statistics to conduct a National Assessment of Educational Progress. The law requires the NCES Commissioner for Education Statistics to conduct a national and state assessment in mathematics and reading in grades four and eight at least once every 2 years. At grade 12, NAEP assesses mathematics and reading at regularly scheduled intervals. Additional national assessments in grades 4, 8, and 12 take place at regularly scheduled intervals in other subjects such as writing, science, history, geography, civics, economics, foreign languages, and arts. This agreement applies to data collection activities under the NAEP program including operational, pilot, feasibility, and special studies.

P.L. 107-110, as amended by P.L. 107-279 authorizes NAEP to include, “whenever feasible, information collected, cross-tabulated, compared, and reported by race, ethnicity, socioeconomic status, gender, disability, and limited English proficiency.” To fulfill this statutory requirement, in addition to cognitive questions, NAEP administers background questionnaires that provide information for reporting categories and that collect non-cognitive data on students, their family background, teachers, and schools.

NCES understands that any improper disclosure or unauthorized use of these materials may violate Federal statutes, including but not limited to the Family Educational Rights and Privacy Act (FERPA) (20 U.S.C. 1232g) as well as applicable state statutes.

By accepting this agreement, NCES acknowledges that all of the provided student records and related information may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S. Code, Section 9573]. Additionally, NCES commits to protect and safeguard these data according to NAEP data security procedures, as applicable, described in NCES Data Confidentiality Procedures Summary (Attachment A) and incorporated herein.

ACCEPTED AND AGREED TO:

_______________________________________ ____________________________________

State authorized agent Date NCES authorized agent Date

_______________________________________ _____________________________________

Title Title

_______________________________________ _____________________________________

Address Address

Appendix K: Sample Communication with Districts Regarding Course Catalog

(NOTE: The communication has been set up as a pair of telephone scripts with a TUDA representative who is not familiar with us. The first script provides the contextual data about their electronic school catalogs and student transcript records, as well as set up the electronic submission of the school catalog. This script can also be modified to be done as an e-mail survey. The survey may be necessary if the TUDA representative does not have the information available and would need to contact another person to get that information, or if the representative does not have time to complete the survey over the phone.)

Hello. My name is <NAME> and I am contacting you from Westat, a research company, which has been contracted by the National Center for Education Statistics of the U.S. Department of Education, to administer the 2017 feasibility study of the NAEP Middle School Transcript Study. Thank you for agreeing to participate in this study. It is the first NAEP transcript study that relies solely on the electronic submission of both course catalogs and student transcripts. The purpose of this phone call is to collect information about the middle school course catalog and student transcripts. This survey should take about 30 minutes, and you may need access to the TUDA’s course catalog and student transcripts to answer the questions. Do you have time now, or would there be a better time to call back?

Now _____ Call back _____

(If Call Back)

What day and time would be best to call back?

(Get call back time, verify that the current telephone number should be used, and end call. For the second call, introduce yourself again and start with the section below.)

__________________________________________________________________________________

(If Now)

Thank you for taking the time to talk with me. Your participation is voluntary. All obtained school and student transcript data may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

First, is there a publicly available online version of your TUDA’s middle school course catalog; that is, a catalog of courses that are available to students in the sixth through eighth grade?

□ Yes □ No

If yes: At what web address can we find this catalog?

________________________________________________________________________________

If no:

Can an electronic copy of the middle school course catalog be provided to us?

□ Yes □ No

If no, please ask for an explanation why there is no TUDA middle school catalog available. Write the explanation below and then skip to Question 4.

________________________________________________________________________________

Note what response was given for this question. It will determine how the script ends.

Does the course catalog provide descriptions for the content of the courses or simply list the available courses?

□ Includes course content descriptions

□ Only lists available courses

For each middle school course, does the course catalog provide the following information:

the number of credits a student earns for passing the course?

□ Yes □ No

a designation in either the course title or course description that notes if the course is a special education course?

□ Yes □ No

a designation in either the course title or course description that notes if the course is an English Language Learner course?

□ Yes □ No

a designation in either the course title or course description that notes if the course is only available as an online course?

□ Yes □ No

a designation in either the course title or course description that notes if the student earns high school credit for passing the course?

□ Yes □ No

Does the catalog provide an identification number for each course?

□ Yes □ No

If yes:

Is this identification number included (or can be included) on the electronic transcripts so that courses from the transcripts can easily be linked to the catalog?

□ Yes □ No

Next, we will ask some questions about the information that can be found on the electronic student transcripts.

Is the following course information identified on the transcripts? If so, how are they indicated (i.e., a field on the record, notation in the course title, etc.)?

Course level (i.e., regular, honors, etc.) □ Yes □ No

____________________________________________________________________________

Course taken off-campus □ Yes □ No

____________________________________________________________________________

Special education course □ Yes □ No

____________________________________________________________________________

English Language Learner course □ Yes □ No

____________________________________________________________________________

High school credit earned for the course □ Yes □ No

____________________________________________________________________________

Course taken online □ Yes □ No

____________________________________________________________________________

Transfer course □ Yes □ No

____________________________________________________________________________

Are the course titles listed on the transcripts the same or abbreviated versions of the course titles listed in the middle school course catalog?

□ Yes □ No

If not, can you explain what differences there are between the catalog course titles and transcript course titles?

____________________________________________________________________________

Next, we will ask some questions on how to interpret data found on the electronic transcripts.

What graduation requirements must a student meet to advance from eighth grade to ninth grade?

____________________________________________________________________________

How many credits did an eighth-grade student earn for a year-long course; that is, a course taken for a single period over the 2016-2017 school year, or its block equivalent?

# of credits, 2016-2017: ____________________

Is the number of credits earned for a year-long course different for either sixth or seventh grade?

□ Yes □ No

If yes:

How many credits did a student in the following grades earn for a year-long course?

# of credits, sixth grade: ____________________

# of credits, seventh grade: ____________________

Has this number of credits earned for a year-long course by sixth through eighth graders changed during the previous two school years?

□ Yes □ No

If yes:

How many credits did a student earn for a year-long course in the following school years?

# of credits, 2014-2015: ____________________

# of credits, 2015-2016: ____________________

What grading system is used in the district for the eighth grade? (Mark one box.)

□ Letter grade (A, B, C, D, etc.)

□ Letter grade with modifiers (A, A-, B+, B, etc.)

□ Pass/Fail

□ Excellent/Satisfactory/Needs Improvement/Unsatisfactory

□ Other (Please specify)

____________________________________________________________________________

Is the grading system different for either sixth or seventh grade?

□ Yes □ No

If yes:

What grading system is used in the district for sixth and seventh grade students?

_______________________________________________________________________

(If a letter grade system is used, continue to Question 10. Otherwise, skip to Question 11.)

What is the numerical range (on a 0 to 100 scale) for each of the letter grades used in the state/district?

Letter Grade or Alternate Symbol	Range (or description, if range not possible)
A+
A
A-
B+
B
B-
C+
C
C-
D+
D
D-
F
Pass
Fail
Excellent
Satisfactory
Needs improvement
Unsatisfactory

Is the grading system the same for all courses, regardless of course level (i.e., special education, honors, Advanced Placement)?

□ Yes □ No

If no

Please explain these differences:

____________________________________________________________________________

Does the state or district require students to complete one or more assessments in order to graduate from sixth, seventh, or eighth grade?

□ Yes □ No

If yes:

Please list these assessments and what grade levels the students take them.

____________________________________________________________________________

Does the district offer any educational programs to students in sixth through eighth grades that focus on their coursework (i.e., offers them courses not normally taught to middle grade students, offers courses that supplements what is normally taught, etc.)?

□ Yes □ No

If yes:

Please list these assessments and what grade levels the students take them.

____________________________________________________________________________

Next, we will ask you about some additional student data that we would like to collect at the same time as the student transcripts. These data are all related to the student’s academic standing and coursework in grades six through eight.

Can the following student information be collected and sent electronically along with the student transcripts?

Eighth grade graduation status (i.e., a variable that indicates whether or not the student graduated from eighth grade at the end of the 2016-2017 school year)

□ Yes □ No

The number of days the student was absent in sixth, seventh, and eighth grades (reported by individual grade level, not a combined count)

□ Yes □ No

Student transfer status (i.e., a variable that indicates whether or not the student transferred into the TUDA school district in sixth, seventh, or eighth grade)

□ Yes □ No

The number of course credits the student earned in sixth, seventh, and eighth grades (reported by individual grade level, not a combined count)

□ Yes □ No

The grade point average the student earned in sixth, seventh, and eighth grades (reported by individual grade level, not a combined count)

□ Yes □ No

If the TUDA can only provide partial data for any of the elements above (e.g., a combined count of absent days or the grade point averages only for seventh and eighth grades), please note it below.

____________________________________________________________________________

If the respondent indicated in Question 1 that there was no on-line course catalog for sixth through eighth grades, but that a catalog could be sent electronically, read the following script. Otherwise, skip to the final paragraph of the script.

Thank you for answering my questions. Finally, I would like to request an electronic copy of the middle school course catalog that you indicated you could send to us. I have several questions about the format of the school catalog and how to send it.

In what format is the school catalog stored? For example, a Microsoft Word file, a Microsoft Excel workbook, ASCII data files, rich text format (RTF) files, PDF files, or some other format.

__________________________________________________________________________________

Will you be encrypting the file containing the school catalog?

□ Yes □ No

If yes:

What encryption program will you be using?

__________________________________________________________________________________

How will you transmit the password to Westat to unlock the encrypted file?

__________________________________________________________________________________

Do you have the ability to access and copy files to an FTP (File Transfer Protocol) web site?

□ Yes □ No

If yes:

Great. Westat will set up a secure FTP site for your state/school district where the school catalog should be downloaded. A follow-up e-mail will contain the web address for your FTP site. To what e-mail address should we be sending this e-mail?

__________________________________________________________________________________

If no:

No problem. Westat will send you a follow-up e-mail, to which you can reply and attach the school catalog. To what e-mail address should we be sending this e-mail?

__________________________________________________________________________________

This is the final paragraph of the first script.

That is all the information we need. We will be calling back during the summer to set up the electronic submission of your student transcripts. Thank you for your time today, and thank you again for agreeing to participate in the Feasibility Study of the NAEP 2017 Middle School Transcript Study. If you have any questions, please feel free to contact <Westat contact name> at <Phone number>. Goodbye. (End call.)

Appendix L: Sample Communication with Districts regarding Student Transcripts

(NOTE: The communication has been set up as a pair of telephone scripts with a TUDA representative who is not familiar with us. This script is a follow-up call to set up the electronic submission of the student transcripts. It can be modified to be done as an e-mail survey. The survey may be necessary if the TUDA representative does not have the information available and would need to contact another person to get that information, or if the representative does not have time to complete the survey over the phone. There will need to be some information transferred over from the first script, mainly the e-mail address of the TUDA representative and whether or not the TUDA representative can access the Westat secure FTP site.)

Hello. My name is <NAME> and I am calling you back again from Westat, a research company on behalf the National Center for Education Statistics of the U.S. Department of Education, about the 2017 feasibility study of the NAEP Middle School Transcript Study (MSTS). Thank you once more for participating in this study. Earlier this year, we called and collected some information about your middle school course catalog and student transcripts. The purpose of this call is to set up the electronic submission of the student transcripts. This call should take about 10 minutes. Do you have time now, or would there be a better time to call back?

Now _____ Call back _____

(If Call Back)

What day and time would be best to call back?

(Get call back time, verify that the current telephone number should be used, and end call. For the second call, introduce yourself again and start with the section below.)

__________________________________________________________________________________

(If Now)

Thank you for taking the time to talk with me. As you know, your participation is voluntary. Any obtained transcript data may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

I would first like to ask some questions about the electronic student transcripts.

Are the electronic transcript data managed and stored within the school district’s data center, or does an outside vendor manage the district’s electronic transcripts?

State/School District Data Center _____ Outside Vendor _____

(If outside vendor)

Would you be the person to contact the vendor to arrange the electronic transfer of the transcripts, or would Westat need to contact them? (If Westat, get the name and telephone number of the vendor contact. Call the contact, use the introduction above, and then start with Question 2.)

__________________________________________________________________________________

In what format is the electronic transcript data files stored? For example, Microsoft Word files, Microsoft Excel workbooks, Microsoft Access databases, SAS databases, SPSS databases, ASCII data files, rich text format (RTF) files, PDF files, or some other format.

__________________________________________________________________________________

Is there personal information about the student listed on the electronic transcripts, such as student name or address?

□ Yes □ No

(If Yes)

Westat can strip out this personal identifying information upon receiving the transcripts, or this information can be stripped out beforehand by you (or your vendor). Which process of removing the student personal information would you prefer?

Westat removes the data _____ School/district/vendor removes the data _____

Will you be encrypting the file containing the transcript data?

□ Yes □ No

(If Yes)

What encryption program will you be using?

__________________________________________________________________________________

How will you transmit the password to Westat to unlock the encrypted file?

__________________________________________________________________________________

If the TUDA posted the school catalog to the secure FTP web site, then continue with the script below. If the catalog was accessed on-line, they sent the school catalog via e-mail, or you are dealing with a software vendor to get the student transcripts, skip to Question 5.

Thank you. As before, we request that you upload the student transcripts to the Westat secure FTP site established for your TUDA. Westat will send you a follow-up e-mail that provides you with the information for the secure FTP site. I have your e-mail address as __filled in from first script_______. Is that correct, or is there another e-mail address should we be sending this e-mail?

__________________________________________________________________________________

Skip to the final paragraph of the script.

Do you have the ability to access and copy files to an FTP (File Transfer Protocol) web site?

□ Yes □ No

(If Yes)

Great. Westat has already set up a secure FTP site for your state/school district where the electronic transcripts file should be downloaded. The follow-up e-mail will contain the web address for your FTP site.

(If No)

Is there anyone in your office who can access and copy files to an FTP web site?

□ Yes □ No

If yes, use the previous script. If no, please note it below, and we will make other arrangements on how to collect the student transcript data.

Thank you. I only need one more piece of information. Westat will send you a follow-up e-mail that provides you with the information for the secure FTP site. I have your e-mail address as __filled in from first script_______. Is that correct, or is there another e-mail address should we be sending this e-mail?

__________________________________________________________________________________

This is the final paragraph of the second script.

Thank you for your time today, and thank you again for participating in the NAEP Middle School Transcript Study. If you have any questions, please feel free to contact <Westat contact name> at <Phone number>. Goodbye. (End call.)

Appendix M: NCES Affidavit of Nondisclosure

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	joc
File Modified	0000-00-00
File Created	2021-01-22