NAEP High School Electronic Transcript Data Collection Feasibility Study

Centralization in a state-level database or regional databases, or all districts have electronic databases
Availability of course completion data elements (enrollment in a specific course, course title, course level [e.g., honors, AP], course grade, course credit, and course code)
Coverage of course subjects (up to six were included in the survey: English, fine arts, foreign language, mathematics, science, and social studies. Note that career and technical education was not included in the survey)
Standardization of course codes (either in national or state-specific classification system)
Availability of auxiliary course information (middle school courses taken for high school credit, courses taken for postsecondary credit, credits transferred from other high schools, AP courses, and other types specified by individual states)
Ability to transfer electronic copies across schools or to the state (any type of transfer)

A combination of values across these six components places a state in one of the four categories: no capacity, minimal capacity, moderate capacity, or strong capacity. A state is placed in categories according to the following rules:

Definitions of state capacity categories
	State capacity categories for electronic transcripts
Data component	None	Minimal	Moderate	Strong
Centralized system	None or only some districts	State, regional, or all districts	State, regional, or all districts	State
Course completion elements	N/A	Four or five	Four or five	All six
Subject availability	N/A	Some	All	All
Standardized codes	N/A	No	Yes	Yes
Auxiliary course elements	N/A	None or one	Two or three	All four
Electronic transfers	N/A	No	No	Yes, any kind

Appendix B: Sample Recruitment Email and Phone Scripts

[Email script:]

Dear [name of state or district official],

My name is <NAME> and I am contacting you from RTI International, an educational research company, which has been contracted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education, to conduct a research study as part of the National Assessment of Educational Progress (High School Transcript Studies program). The purpose of the study is to better understand how states, districts, and schools store and transfer high school transcript data electronically.

We would like to talk to you about [name of state or district]’s use of electronic high school transcripts. Would you be willing to meet by phone for an interview—or suggest another person who would be best qualified to discuss this? You may respond directly to this email or call at my number below. Your participation is voluntary.

Thank you,

[Name of RTI International staff member]

[Direct phone line of RTI International staff member]

[Phone script:]

If contact is willing to participate or recommend another potential participant gather the appropriate information. If not, thank the contact for his/her time.

Appendix C: Guiding Questions for Semi-structured Interview with State or District Officials

Thank you for agreeing to talk with us about the Electronic Transcript Data Collection Feasibility Study. My name is <NAME> and I am contacting you from RTI International, an educational research company, which has been contracted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education, to conduct a research study as part of the National Assessment of Educational Progress (High School Transcript Studies program). The purpose of the study is to better understand how states, districts, and schools store and transfer high school transcript data electronically. Your participation is voluntary.

I would like to talk to you about [name of state or district]’s use of electronic high school transcripts.

Do you have any questions? Let’s begin.

How are your [state’s/district’s] high school transcripts stored electronically?

Are coursetaking records arranged by year or incorporated into multi-year, student-based transcript records?
Is third-party software used to manage or transfer transcript records (e.g., National Transcript Center, Parchment)?
What eTranscript standards or formats are used for transfer to other agencies or schools (e.g., XML/PESC, SPEEDE EDI, SIF eTranscript)?

What course coding system or systems are used for your transcript data (e.g., state/local-specific system, CSSC, SCED)?
What documentation exists concerning course catalog information either at your [state/district] level or at other levels (state, district, or school)?

In what form is that documentation (e.g., Microsoft Word files, PDF files, Microsoft Excel workbooks, a database)?

What are the policies governing research access to [state/district] electronic transcript records?

Does your [state/district] allow access or permit transfer of student records to a federal agency such as the National Center for Education Statistics (NCES))?
Would a data use agreement be required to share data with NCES?
What is the application process and typical timeline for establishing a data use agreement?
Are there additional research applications or IRB (Institutional Review Board) requirements for obtaining electronic transcript data?

Are you aware of examples of the [state/district] working with federal groups to share data? If so, can you describe those efforts?
How do you think your [state/district] would view efforts to share electronic high school transcript records with NCES? What potential benefits would you see in such sharing? What potential drawbacks would you see?

Thank you for your participation. RTI International will provide you with a summary of your responses by email to allow for any corrections or clarifications, should you wish to provide them.

Appendix D: Sample Thank You Email/Letter

[(For letters only:)

Name of RTI International staff member

3040 Cornwallis Rd

PO Box 12194

Research Triangle Park, NC 27709]

[(For letters only:)

Name of respondent

Respondent address]

Dear [name of state or district respondent and/or their designee(s)],

Thank you for discussing [name of state’s/district’s] electronic transcript systems. The information you supplied was very helpful. Enclosed is a summary of our interview. Please feel free to send any corrections or clarifications to the summary to us. If you have none, then nothing further is needed.

Sincerely,

[Name of RTI International staff member]

Appendix E: Sample Communication with the State/School District Coordinators

Telephone Script for Interviewing State Data Center Contact/District Data Manager for Obtaining Electronic Transcripts

Hello. My name is <NAME> and I am contacting you from Westat, an educational research company, which has been contracted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education, to conduct the NAEP High School Transcript Study (HSTS) Electronic Transcript Feasibility Study. Several months ago, your state/school district was contacted by RTI International for the first part of the study, which examined how states, districts, and schools store and transfer high school transcript data electronically. Your name was given to us as a person to contact about obtaining the electronic transcripts. Today’s call covers the second part of the study, which involves contacting those states and districts capable of transferring high school transcript data electronically. We are collecting a sample of electronic high school transcripts from schools across the country to see if NCES can improve the quality of the data collection for future transcript studies. These transcripts would be collected as part of the 2015 NAEP grade 12 assessments under the auspices of the NAEP High School Transcript Studies program.

We have been given your name as the state/school district contact person who is able to arrange and authorize the electronic transfer of high school transcripts for the NAEP High School Transcript Study (HSTS) Electronic Transcript Feasibility Study. Is this correct?

Yes _____ No _____

(If No)

Who would be the correct person to contact? (Get name and telephone number of the new contact and end interview.)

__________________________________________________________________________________

(If Yes)

Would you be willing to discuss the study now, or would there be a better time when I should call you back?

Now _____ Call back _____

(If Call Back)

What time would be best to call back?

(Get call back time, verify that the current telephone number should be used, and end call. For the second call, introduce yourself again and start with the section below.)

__________________________________________________________________________________

(If Now)

Thank you for taking the time to talk with me. Your participation is voluntary. Any obtained transcript data may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

We would like to obtain a sample of electronic high school transcripts from the twelfth-graders in your high schools that were sampled for the NAEP 2015 assessments. As part of the feasibility study, we want to examine the various electronic transcript formats used across the nation and see if the electronic data on the transcripts are sufficient to be able to conduct future transcript studies. As with all NAEP studies, the names of schools and students participating in the study would not be included in any of our reports. Would you be willing to participate in this study?

Yes _____ No _____

(If No)

Thank you very much for your time. (End call.)

(If Yes)

Thank you for participating in our study. We are looking to obtain a sample of 25 electronic high school student transcripts for a number of high schools in your state/district that participated in the NAEP 2015 Grade 12 assessments. The number of schools from which we want transcripts will be no more than 10 schools (if state)/three schools (if district). We will pay the state/district two dollars for each collected transcript. We will send more detail about the sampled schools and students in a follow-up e-mail.

For this interview, we need to gather more information about the electronic transcripts.

Are the electronic transcript data managed and stored within the state /school district data center, or does an outside vendor manage the state’s/district’s electronic transcripts?

State/School District Data Center _____ Outside Vendor _____

(If outside vendor)

Would you be the person to contact the vendor to arrange the electronic transfer of the transcripts, or would Westat need to contact them? (If Westat, get the name and telephone number of the vendor contact.)

__________________________________________________________________________________

In what format is the electronic transcript data files stored? For example, Microsoft Word files, Microsoft Excel workbooks, Microsoft Access databases, SAS databases, SPSS databases, ASCII data files, rich text format (RTF) files, PDF files, or some other format.

__________________________________________________________________________________

Is there personal information about the student listed on the electronic transcripts, such as student name or address?

Yes _____ No _____

(If Yes)

Westat can strip out this personal identifying information upon receiving the transcripts, or this information can be stripped out beforehand by you (or your vendor). Which process of removing the student personal information would you prefer?

Westat removes the data _____ School/district/vendor removes the data _____

(If school/district/vendor)

In future NAEP transcript studies, when trying to match student transcript data with their NAEP assessment results and questionnaires, we require a cross-walk system that allows us to link the NAEP and transcript data together. This cross-walk system is a secure system to protect the identities of the sampled students. We will not be implementing a cross-walk system for the feasibility study, but we wanted to offer the opportunity to discuss how such cross-walk systems can be implemented. Would you like additional information about cross-walk systems sent in the follow-up e-mail?

Yes _____ No _____

Will you be encrypting the file containing the transcript data?

Yes _____ No _____

(If Yes)

What encryption program will you be using?

__________________________________________________________________________________

How will you transmit the password to Westat to unlock the encrypted file?

__________________________________________________________________________________

Do you have the ability to access and copy files to an FTP (File Transfer Protocol) web site?

Yes _____ No _____

(If Yes)

Westat is setting up a secure FTP site for your state/school district where the electronic transcripts file should be downloaded. The follow-up e-mail will contain the web address for your FTP site.

Do you have knowledge of the content of the transcript; that is, the information stored on the transcript, like course name, credits earned, and grade earned?

Yes _____ No _____

(If Yes, continue the interview with the questionnaire located in appendix G.)

(If No)

Who would be the best person to contact about the transcript contents? (Get name and telephone number of the new contact.)

__________________________________________________________________________________

Thank you. I only need one more piece of information. Westat will send you a follow-up e-mail that verifies your participation and provides more detail about the schools selected for the study, how to sample the students from those schools, and the secure FTP site where to download the student transcripts. To what e-mail address should we be sending this e-mail?

__________________________________________________________________________________

Thank you for your time today, and thank you again for participating in the NAEP High School Transcript Study Electronic Transcript Feasibility Study. (End call.)

Appendix F: NAEP 2015 Grade 12 State Sample

Number of sampled schools and students in the NAEP 2015 Grade 12 state sample

State	Number of sampled schools	Number of sampled students
AK	3	278
AL	7	615
AR	11	761
AZ	24	1,871
CA	21	1,995
CO	7	605
CT	11	1,045
DC	2	109
FL	33	3,079
GA	21	1,995
IA	5	394
ID	2	157
IL	25	2,197
IN	7	618
KS	8	510
KY	10	828
LA	10	862
MA	7	653
ME	2	123
MI	21	1,550
MN	12	922
MO	15	1,330
MS	8	560
NC	20	1,746
NE	1	48
NH	3	285
NJ	16	1,404
NM	7	449
NV	10	875
NY	42	3,447
OH	25	2,154
OK	8	625
OR	7	546
PA	23	2,151
SC	13	1,244
SD	1	5
TN	14	1,315
TX	75	6,307
UT	7	590
VA	20	1,900
WA	18	1,602
WI	14	1,109
WV	3	239
WY	2	113
Total	601	51,211

Appendix G: Communication with State/School District Education Data Center

NAEP 2015 ELECTRONIC TRANSCRIPT STUDY

SCHOOL INFORMATION FORM QUESTIONS

(If the person being interviewed is the same person who was interviewed for obtaining electronic transcripts, proceed to question 1. If it is a different person, start with the script below.)

Hello. My name is <NAME> and I am contacting you from Westat, an educational research company, which has been contracted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education, to conduct the NAEP High School Transcript Study (HSTS) Electronic Transcript Feasibility Study. The study examines how states, districts, and schools store and transfer high school transcript data electronically and if electronic transcripts can be used to improve the data collection for future transcript studies, such as the NAEP High School Transcript Study. Your state/school district is participating in the study by sending a sample of electronic student transcripts from high schools that were sampled for the 2015 NAEP assessments. Your name was given to us as a person knowledgeable about the data stored on the electronic transcripts. Would you be willing to discuss the electronic transcript data now, or would there be a better time to call back?

Now _____ Call back _____

(If Call Back)

What time would be best to call back?

(Get call back time, verify that the current telephone number should be used, and end call. For the second call, introduce yourself again and start with the section below.)

__________________________________________________________________________________

(If Now)

First, we will discuss the availability of your state/district high school catalog. Is there a publicly available online version of your state/district high school course catalog; that is, a catalog of courses that are available to students in the 9^th through 12^th grades?

□ Yes □ No

If yes: At what web address can we find this catalog?

________________________________________________________________________________

If no:

Can an electronic copy of the high school course catalog be provided to us?

□ Yes □ No

If no:

Can a physical copy of the high school course catalog be provided to us?

□ Yes □ No

If no, please ask for an explanation why there is no state/district high school catalog available. Write the explanation below and then skip to Question 4.

________________________________________________________________________________

Does the high school course catalog provide descriptions for the content of the courses or simply lists the available courses?

□ Includes course content descriptions

□ Only lists available courses

Does the catalog provide an identification number for each course?

□ Yes □ No

If yes:

Is this identification number included (or can be included) on the electronic transcripts so that courses from the transcripts can easily be linked to the catalog?

□ Yes □ No

Now, we will ask some questions about the information that can be found on the electronic transcripts.

Is the following course information identified on the transcripts? If so, how are they indicated (i.e., a field on the record, notation in the course title, etc.)?

Special education course □ Yes □ No

____________________________________________________________________________

Course taken off-campus □ Yes □ No

____________________________________________________________________________

Course taught in language other than English □ Yes □ No

____________________________________________________________________________

College credit earned for the course □ Yes □ No

____________________________________________________________________________

Course taken online □ Yes □ No

____________________________________________________________________________

Transfer course □ Yes □ No

____________________________________________________________________________

Are the course titles listed on the transcripts the same as the course titles listed in the course catalog?

□ Yes □ No

If not, can you provide an explanation on the differences between the catalog course titles and transcript course titles? For example, is there a list of common abbreviations used on the electronic transcripts that could be provided?

____________________________________________________________________________

Finally, we will ask some questions on how to interpret data found on the electronic transcripts.

How many credits did a student earn for a year-long course; that is, a course taken for a single period over the 2014-2015 school year, or its block equivalent?

# of credits, 2014-2015: ____________________

Has this number of credits changed during the previous three school years?

□ Yes □ No

If yes, how many credits did a student earn for a year-long course in the following school years?

# of credits, 2011-2012: ____________________

# of credits, 2012-2013: ____________________

# of credits, 2013-2014: ____________________

What grading system is used in the state/district? (Mark one box.)

□ Letter grade (A, B, C, D, etc.)

□ Letter grade with modifiers (A, A-, B+, B, etc.)

□ Pass/Fail

□ Satisfactory/Unsatisfactory

□ Other (Please specify)

____________________________________________________________________________

(If a letter grade system is used, continue to Question 8. Otherwise, skip to Question 9.)

What is the numerical range (on a 0 to 100 scale) for each of the letter grades used in the state/district?

Letter Grade or Alternate Symbol	Range (or description, if range not possible)
A+
A
A-
B+
B
B-
C+
C
C-
D+
D
D-
F
Pass
Fail
Satisfactory
Unsatisfactory

Is the grading system the same for all courses, regardless of course level (i.e., special education, honors, Advanced Placement)?

□ Yes □ No

If no, please explain these differences:

____________________________________________________________________________

(If the person being interviewed is the same person who was interviewed for obtaining electronic transcripts, return to question 9 on appendix E. If it is a different person, continue with the script below.)

That is all the information we need. Thank you for your time today, and thank you again for participating in the NAEP High School Transcript Study Electronic Transcript Feasibility Study. (End call.)

NOTE: If there are items that need to be sent to us (electronic copy of course catalog, physical copy of course catalog, transcript abbreviations list, etc.), we will need to provide the appropriate email or mailing address. That information should be shared at the end of the interview.

Appendix H: Minimum Content Requirements for Grade 12 Electronic Transcripts

Appendix J: NCES Affidavit of Nondisclosure

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	0000-00-00