TOC Supplement

Partners in the Data Quality Initiative

To: Bonnie Jones, Office of Special Education and Rehabilitative Services (OSERS)

Date: June 1, 2011

From: Patty Troppe, Data Quality Initiative (DQI)

cc: Melanie Ali, Institute of Education Sciences

RE: A review of data collection approaches for GPRA measures that require information on participants after the end of their project services--DQI programs only

This memo provides a summary of data collection procedures for Government Performance Results Act (GPRA) measures that require information on participants after the end of their project services. The memo is limited to those ED Program Offices that have received technical assistance through the DQI contract with IES.¹ This memo is the result of a request from OSERS to learn how other programs collect placement, performance, and retention information on graduates of teacher programs as it moves forward with its data collection for the GPRA measures for the Personnel Development Program (PDP).²

This memo provides information from the four DQI programs with GPRA measures that require the collection of information on participants (who may be teachers or other individuals) after the end of their project services. Some of the measures align closely

with those of the OSERS Personnel Development Program; however, none of the measures have the exact same language. We present information on the data collection processes for all four programs given the differences in data collection approaches.

This memo begins with a brief overview of each program and identifies their GPRA measures. Following the overview is a set of “data collection questions” that ask about core elements of the data collection process and responses to those questions looking across the four programs. For example, who is responsible for tracking participants, which participants are followed? The responses provide a summary of how the different programs implement particular parts of their data collections.

The memo ends with the question, “What information is available regarding participant follow- up rates?” At this time, there is limited information available. There is only one Program Office with some information on participant follow up. For two Program Offices, grantees are just starting to report on the measures. The Program Offices will learn about the extent of participant follow up as part of grantee reporting on the measures. For another Program Office, grantees and the Program Office staff are implementing the first round of data collection using new forms in spring 2011. Information on participant follow up should be available when the Program Office reports on the measures.

Program Overviews and GPRA Measures

This section provides a brief overview of the four relevant DQI programs. Table 1 lists the GPRA measures for each program.

• The Office of Indian Education (OIE) Professional Development (PD) program funds grantees to prepare and train Indians to serve as teachers and administrators, who then are expected to work in a target local education agency (LEA). A target LEA is defined as one that enrolls at least five percent Native American or Alaska Native (NA/AN) students. Participants may receive training in teacher or administrator tracks. Participants must complete a service payback requirement by either (1) working in a target LEA in the field in which he or she trained under the project for an amount of time commensurate with the amount of training received, or (2) through cash payback to ED. Participants may receive diplomas, certificates, or a combination of diploma and certificate, depending on the nature of the project. Grantees are typically Institutions of Higher Education (IHEs), but State Education Agencies (SEAs), LEAs, and Indian tribes or organizations can be grantees in consortium with IHEs.

This program has six GPRA measures. All measures require information about participants after the end of their project services. Grantees do not report on the measures or follow up with participants after they exit project services. Instead, grantees provide the Program Office with information about each participant’s training and contact information so it can follow-up with participants. Grantees and the Program Office are using new, custom data collection forms for the first time in spring 2011.

Table 1. GPRA Measures for DQI programs with measures that require the collection of information on participants after the end of their

project services

Office and Program	GPRA Measures
OIE-Professional Development Program	Measure 1: The percentage of participants in administrator preparation projects who become principals, vice principals, or school administrators in local educational agencies (LEAs) that enroll 5 percent or more American Indian and Alaska Native students. Measure 2: The percentage of participants in teacher preparation projects who become teachers in LEAs that enroll 5 percent or more American Indian and Alaska Native students. Measure 3: The percentage of program participants who meet the definition of “Highly Qualified” in section 9101(23) of the ESEA. Measure 4: The percentage of program participants who complete their service requirement on schedule. Measure 5: The cost per individual who successfully completes an administrator preparation program, takes a position in a school district with at least 5 percent American Indian/Alaska Native enrollment, and completes the service requirement in such a district. Measure 6: The cost per individual who successfully completes a teacher preparation program, takes a position in a school district with at least 5 percent American Indian/Alaska Native enrollment, and completes the service requirement in such a district.
OELA- National Professional Development Program	Measure 1.1: The percentage of pre-service program completers who are state and/or locally certified, licensed, or endorsed in LEP instruction. Measure 1.2: The percentage of pre-service program completers who are placed in instructional settings serving LEP students within 1 year of program completion. Measure 1.3: The percentage of pre-service program completers who are providing instructional services to LEP students 3 years after program completion. (This measure applies to grants awarded in 2010 onward). Measure 1.4: The percentage of paraprofessional program completers who meet state and/or local qualifications for paraprofessionals working with LEP students. Measure 1.5: The percentage of in-service teacher completers who complete state and/or local certification, licensure, or endorsement requirements in LEP instruction as a result of the program. Measure 1.6: The percentage of in-service teacher completers who are providing instructional services to LEP students.

Table 1. GPRA Measures for programs with measures that require the collection of information on participants after the end of their

project services (continued)

Office and Program	GPRA Measures
OII-School Leadership Program	Measure 1.1: The percentage of School Leadership Program participants seeking certification who meet certification requirements to become a principal or assistant principal. Measure 1.2: The percentage of School Leadership Program participants certified through the funded project who are hired as an assistant principal of a school in a high-need LEA. Measure 1.3: The percentage of School Leadership Program participants certified through the funded project who are hired as a principal of a school in a high-need LEA. Measure 1.4: The percentage of School Leadership Program participants who are hired as a principal or assistant principal of a school in a high-need LEA and who remain in that position for at least 2 years. Measure 2.1: The percentage of principals and assistant principals from schools in high-need LEAs who participated in the School Leadership Program-funded professional development activities who showed an increase in their pre-post scores on a standardized measure of principal skills. Measure 2.2: The percentage of principals and assistant principals from schools in high-need LEAs who participated in the School Leadership Program-funded professional development activities who remained in their administrative position for at least 2 years.
OME-High School Equivalency Program	Measure 1.1: The percentage of HEP program exiters receiving a General Educational Development (GED) diploma. Measure 2.1: The percentage of HEP GED recipients who enter postsecondary education or training programs, upgraded employment, or the military.

• The Office of English Language Acquisition (OELA) National Professional Development Program (NPD) funds grantees to provide professional development activities intended to improve instruction for students with limited English proficiency (LEP) and assist education personnel working with such children to meet high professional standards. Grantees may provide services to pre-service teachers, in-service teachers, and paraprofessionals. NPD grants are awarded to IHEs in consortia with LEAs or SEAs.

NPD has six GPRA measures. Four of the measures require information on program completers as of the end of the reporting period. However, two of the measures require longer follow-up information about program completers after graduation. Measure 1.2 requires placement information for pre-service program completers within 1 year of graduation. Measure 1.3 requires placement information for pre-service completers 3 years after graduation. Measure 1.3 applies only to grants awarded in 2010 onward. However, participant follow up does not extend beyond the period of the grant, so it is possible that grantees may not follow up with all participants for Measures 1.2 and 1.3.

• The Office of Innovation and Improvement (OII) School Leadership Program (SLP) provides grants to assist high-need LEAs with recruiting, training, and retaining principals and assistant principals. A high-need LEA is defined as one that: (1) serves at least 10,000 children from low-income families or serves a community in which at least 20 percent of children are from low-income families and (2) has a high percentage of teachers teaching either outside of their certification or with emergency, provisional, or temporary certification.

The program has six GPRA measures. Four measures relate to a program objective to recruit, prepare, and support individuals from education or other fields to become principals or assistant principals of schools in high-need LEAs. The other two measures relate to a program objective to train and support principals and assistant principals from schools in high-need LEAs in order to improve their skills and increase retention. Many of the measures require employment information on participants after the end of their project services, including Measures 1.4 and 2.2 that require information about retention in a position after 2 years. The cohort of FY 2008 grantees will report on the retention measures for the first time in summer 2011. Like NPD grantees, SLP grantees are not expected to follow up with participants beyond the period of the grant, so it is possible that grantees will not have data on all participants for Measures 1.4 and 2.2.

• The Office of Migrant Education (OME) High School Equivalency Program (HEP) is intended to assist migrant and seasonal farmworker students in obtaining the equivalent of a high school diploma and, subsequently, to begin postsecondary education, enter military service, or obtain employment. Grantees are generally IHEs, but a public or nonprofit private agency in cooperation with an IHE may also apply for funding.

There is one GPRA measure related to the objective of increasing the percentage of HEP participants who receive their General Educational Development (GED) diploma (Measure 1.1). There is a second measure related to the other objective of increasing the percentage of HEP recipients of the GED who enter postsecondary education or training programs, upgraded employment, or the military (Measure 2.1). HEP grantees follow up with participants only until the annual reporting deadline.

Data Collection Questions

This section provides detailed information on the data collection approaches used by the four DQI programs. The section is in a question and answer format so the reader can identify a particular part of the data collection approach and see related information for the four programs.

Who is responsible for following up with participants?

All grantees, except for OIE-PD grantees, are responsible for tracking and following up with participants after the end of their project services. The frequency of follow up depends on the measure. For some measures, the grantee only needs to follow up with a participant through the end of the reporting period. For other measures, the grantee needs to follow up with a participant 1, 2, or 3 years after the participant ends project services. However, none of these grantees is expected to follow up with participants after the grantee period of performance ends.

OIE-PD grantees are responsible for collecting contact and project services/training information for each participant. However, this responsibility ends when the participant exits project services. Instead, the OIE-PD Program Office staff are responsible for tracking participants after the end of their project services until a participant fulfills service payback. The OIE-PD Program Office staff sends a letter to participants who have exited project services informing them of their service obligation and requirement to stay in touch with the Program Office. Program Office staff send this letter as soon as they hear from grantees that a participant has exited project services. However, the Program Office staff are responsible for following up with any participant who does not contact the Program Office in a timely manner.

Which participants are followed?

Not all Program Offices require follow up with each participant after the end of their project services to report on GPRA measures. While the OIE-PD Program Office is required to follow up with all participants because of the service payback agreement, the HEP Program Office allows grantees to draw a sample of participants for follow up for Measure 2.1 if the grantee meets certain conditions (described below).

The NPD and SLP Program Offices encourage grantees to follow up with all participants after the end of their project services. However, for some GPRA measures, the number of participants that can be followed is a function of several factors including the length of the participant’s services, the amount of time between the end of the individuals’ project services and the performance outcome of interest (e.g., retention 2 years after graduation), and the grantee’s period of performance.

Finally, for measures where the grantee follows a participant over a year or more, different Program Offices have different rules about whether the grantee should follow a participant that “leaves the area” (e.g., is no longer employed in the grantee’s district).

• OIE-PD participants enter into a service payback agreement as a condition of project services. Given this agreement, the Program Office follows up with every participant to gather information for the GPRA measures and to determine if the participant has fulfilled service payback. If the Program Office cannot locate an individual after repeated attempts, the individual’s contact information and Social Security Number are turned over to ED’s debt services department.

• The HEP Program Office encourages grantees to follow up with all participants to determine if they received a GED for Measure 1.1. However, grantees with a large number of GED attainers may draw a sample for follow up for reporting on Measure 2.1. Specifically, the Program Office allows grantees with at least 150 GED attainers in a project year to draw a scientific sample for follow up, subject to Program Office approval. Grantees that receive approval to sample must draw a scientific sample using a DQI-developed sampling spreadsheet and instructions. Projects with 149 or fewer GED attainers in the project year must conduct follow-up with all GED attainers.

• For the NPD and SLP placement and retention measures, the number of participants that should be followed and reported on is a function of the interaction between the grant length, the participation period for a given individual, and the follow-up period necessitated by the measure. For example:

• NPD grants are awarded for up to 5 years.  The placement and retention measures (Measures 1.2 and 1.3) pertain to programs serving pre-service participants.  Grantees are not required to follow up with participants after the grant award ends so the year of study participants are in at the start of the grant is important.  For example, if a participant was in his or her first year of a four-year bachelor’s degree program, the grantee would only be able to follow up with that participant after 1 year of completing the program of study, not 3 years.

• SLP grants are 5 years.  Grantees serve multiple cohorts of participants, and they report on participants separately by the year participants started in the program.  For Measure 1.4 on retention after 2 years, by the end of their grant cycle, the grantee can only include participants who were hired in the first, second, or third project years. The grantee would not be able to report on any participant who was hired in the fourth or fifth project year in this measure. For Measure 2.2, if a grantee has a multi-year project that lasts 4 or 5 years and it will not be able to report on participants 2 years after graduation or completion, the grantee is asked to report to the Program Office the number of participants and the number who are still employed in the same high-need LEA as an assistant principal or principal as of the end of the grant. This information is not captured within the GPRA measure, however.

• The NPD and SLP Program Offices differ in their expectations regarding grantees following up with participants who leave the area. For NPD Measure 1.3, grantees are expected to follow up with pre-service program completers who have left the district(s) served by the grant. The SLP Program Office does not have such a requirement.

Do grantees receive any guidance on tracking participants?

Of the three programs that require grantees to track participants, only the HEP Program Office provides grantees with formal guidance for how to track participants after the end of their project services. The HEP Program Office distributes to grantees a DQI-developed guidance document that identifies and describes activities to collect this information for GPRA reporting, including: collecting participant contact information; determining a data collection method; implementing the data collection; and entering and managing follow-up data. The attachment to this memo includes a copy of this document.

What sources of information are used to document whether participants are meeting the GPRA measure outcome of interest?

Information on outcomes from official sources (e.g., employment records) is a more valid and reliable way to document whether a participant is meeting an outcome of interest than relying on participant self report. Some Program Offices are very specific on what sources to use, while others do not provide specific guidance on this.

• The new data collection system for OIE-PD involves three OMB-approved data collection forms to collect the needed data.³ Participant contact and project services/training information (e.g., field of training, number of semesters of training) are collected from grantees every 6 months using a Semi-Annual Participant Report (SAPR). The participant is expected to contact the Program Office within 6 months of exiting a project and provide employment information or to make a request for an authorized deferment. However, if this does not happen, the Program Office will initiate contact with the participant, using the contact (or alternate contact) information provided on the SAPR submitted by grantees. The Program Office uses a Participant Follow-Up Protocol to collect participants’ current contact information, education enrollment status⁴, and employment information including the names and addresses of schools and LEAs if the participant is employed as a teacher or administrator. Participant follow up will occur every 6 months until the participant has completed either the service or the cash payback requirement.

The Program Office will send the Employment Verification Form to participants intending employment as a teacher or administrator and those already employed in these positions. Participants must give an Employment Verification Form to the principal or LEA representative under whom they work to verify the nature and duration of their employment. The participant must initiate the employment verification process with his or her principal or LEA representative within 6 months of exiting project services and every 6 months thereafter until the end of the participant’s service obligation. Thus, follow-up data collection will be ongoing and could occur throughout the year depending on when a participant exits the project. If a participant worked at multiple schools and/or in multiple positions at schools since exiting project services, the participant must provide an Employment Verification Form to each principal or LEA representative. The principal or LEA representative sends the completed and signed Employment Verification Form directly to the OIE-PD Program Office.

With the data from Employment Verification Form, the Program Office verifies the participant’s contact and identifying information (i.e., date of birth and last four digits of participant Social Security Number) and employment information: position (teacher or administrator), employment duration, certification, and for teachers, information on subject(s) taught and Highly Qualified Teacher status. ED also requests on the Employment Verification Form the percentage of American Indian/Alaska Native students enrolled in the LEA for the most recent school year. The Program Office uses this information to determine whether a participant is meeting the outcomes of interest for the GPRA measures and whether participants are fulfilling service obligation requirements.

• For SLP, grantees are either high-need LEAs or IHEs partnering with high-need LEAs, so they will use the LEA’s employment records. SLP grantees will verify if

a participant remained in their administrative position 2 years after their initial hiring for Measure 1.4 or after completion of professional development for Measure 2.2. The participant does not need to be in the same school during that time period, but must remain in the same high-need LEA.

• The NPD Program Office does not instruct grantees about what sources to use to verify participant placement after project services end. As a result, the sources may vary by grantee.

Also of note is that for Measure 1.3 (which measures placement 3 years after graduation), grantees need to determine whether the program completer is providing instructional services to LEP students only at one point in time—during the reporting period 3 years after program completion. Pre-service program completers do not need to provide instructional services to LEP students for 3 consecutive years to be counted as meeting the measure.

• For HEP, the Program Office guidance to grantees on collecting follow-up information on participants recommends that grantees obtain official documentation from participants to determine if participants entered postsecondary education, upgraded employment, or the military. Examples of official documentation include a copy of a pay stub or registration confirmation information from a postsecondary institution. Another method identified in the guidance is for grantees to enter into a data matching agreement with another institution. With data matching, the grantee provides participant identifying information to the institution, which, in turn, searches for the participant in its databases. For example, a grantee may obtain postsecondary education enrollment information from a local college. Data matching also can be used to supplement telephone contacts or questionnaires, particularly if contacts/response rates are low.

What is the frequency of participant follow up?

• For OIE-PD, grantees will update contact and project services information and submit this to the Program Office every 6 months during the grant period of performance. Participants are supposed to contact the Program Office within 6 months of exiting project services, and continue to initiate this contact every 6 months until they fulfill their service payback. However, if the participant does not initiate contact with the Program Office, then the Program Office will initiate contact in 6-month intervals.

• For HEP, the Program Office provides some guidelines for frequency of follow up in the grantee guidance document. In general, GED attainers are expected to achieve their post-GED placement within 4 weeks of attaining the GED. This, however, may not be the case for every participant, and in some cases, ongoing contact with GED attainers may be necessary. The Program Office recommends that grantees start attempting follow-up with their GED attainers as soon as possible but no later than 4 weeks after the attainer leaves the project. Once the grantee has achieved follow-up for a GED attainer and the attainer has achieved a desired post-GED placement (i.e., enrolled in postsecondary education, upgraded employment, or joined the military) the GED attainer does not need to be contacted for this information again. For those GED attainers who could not be contacted during the first attempt or did not have a desired post-GED placement at the time of the first contact, the Program Office recommends the grantee continue to attempt follow-up to collect their placement data, up until the project’s annual internal reporting deadlines.

• NPD grantees follow up with pre-service program completers twice: once within 1 year of graduation and again 3 years after program graduation.

• SLP grantees will need to gather information on a participant’s current employment status one time.  For Measure 1.4, the grantee will gather information 2 years after a participant is hired.  For Measure 2.2, the grantee will gather information 2 years after a participant completes professional development.

What information is available regarding participant follow-up rates?

For three programs, there is no information available yet regarding participant follow-up rates. However, there is some information on the HEP measure.

• For HEP, the Program Office indicates that grantees are conducting participant follow up for Measure 2.1. The Program Office suggests that the grantee-level calculation of Measure 2.1 provides an incentive for grantees to follow up with participants. This measure is calculated by dividing the number of GED attainers who entered postsecondary education or training programs, upgraded employment, or the military by the total number of GED attainers. Note that the denominator is not the number of GED attainers followed, but rather all GED attainers (or, for those grantees allowed to sample, the number of GED attainers sampled ). Therefore, if a grantee does not follow up with a GED attainer to determine post-project placement, the participant is considered for the calculation as not meeting the desired post-placement outcome. This calculation might lead to a lower grantee percentage for this measure than would occur if the grantee contacted more participants. Since the Program Office considers grantee-level GPRA measure findings when considering grantee continuation funding, the Program Office indicates that grantees have motivation for following up with participants.

NPD grantees will provide information about the number of participants followed as part of their reporting on placement measures. SLP grantees will describe any missing participant data as part of their reporting on the retention measures, which should include information on the number of participants not followed. The OIE-PD Program Office staff will be able to report on the number of participants it follows over time when it reports on its measures. The following provides a timeline for reporting on these measures.

• Grantees will report on the SLP 2-year retention measures for the first time in summer 2011.

• The NPD 3-year retention measure applies to FY 2010 grantees and onward, so it will be several years before grantees report on this measure.

• The OIE-PD Program Office just instituted a new data collection approach with grantees providing their first submission of participant data at the end of May 2011. The Program Office then needs to follow up with those participants who have exited project services within the next 6 months, and verify employment. It will likely be another year before OIE-PD will start reporting on many of the GPRA measures.

Attachment

Guidance to the Office of Migrant Education High School Equivalency Program Grantees for Collecting Participant Follow-Up Data

Partners in the Data Quality Initiative

Guidance to the Office of Migrant Education High School Equivalency Program Grantees for Collecting Participant Follow-Up Data

Authors
Sarah Heinemeier, Compass

Patty Troppe, Westat

March 24, 2009

Prepared for:
Office of Migrant Education
U.S. Department of Education

Introduction

The Office of Migrant Education (OME) High School Equivalency Program (HEP) grantees are required to report on two GPRA measures. This document provides guidance to grantees on follow-up activities necessary to report on GPRA Measure 2:

The percentage of HEP GED recipients who enter postsecondary education programs, upgraded employment, or the military.

This performance measure requires grantees to collect follow-up data on all participants who attained a General Educational Development (GED) credential (i.e., GED attainers) to determine if they entered postsecondary education, upgraded employment, or the military.⁵ These data also will inform project and program office monitoring and evaluation questions. All data should be reported on the OME HEP customized Annual Performance Report (APR).

Follow-up should be conducted on all GED attainers who exited the program in the project year reported in the December 31^st report.

This document identifies and describes activities to collect this information for GPRA reporting, including:

• collecting participant contact information;

• determining a data collection method;

• implementing the data collection; and

• entering and managing follow-up data.

The guidance presented is based in part on materials produced by the U.S. Department of Education’s Office of Vocational and Adult Education, National Reporting System (NRS⁶), as well as information and discussion generated by the Education Department’s Data Quality Initiative. This guidance is supplemented with an appendix containing additional references on survey development and implementation.

Collecting participant contact information

Maintain multiple contact numbers or addresses for each participant

Participants should be notified that they will be contacted after exiting the program to provide information on their post-GED education, employment, or military placement status.⁷ For the migrant population served with this program, it is important to collect, if possible, multiple telephone contact numbers and/or addresses that will be viable after the GED attainer has exited the project. These should include phone numbers or contact information of relatives or others who will know how to contact the participant. This information should be maintained in a database (or spreadsheet) and be periodically updated while the participant is receiving project services.

Determining a data collection method

There are multiple methods for collecting data, including written questionnaires (such as questionnaire completed on site or disseminated via the U.S. Post, email or with Web-surveys), telephone interviews, individual face-to-face interviews, and data matching.⁸ These methods also can be used in combination. We anticipate that multiple data collection methods will be necessary to contact all GED attainers.

Consider cost and representation when choosing a data collection method.

There are several factors to consider in choosing a data collection method, including cost and representation of target population. Table 1 summarizes some of the considerations attached to various data collection methods.

Table 1: Considerations of OME HEP Data Collection Methods

Data Collection Method	Considerations
Written Questionnaires Completed On Site	On-site questionnaires are easy to administer BUT the data will not represent all GED attainers--only those on site who complete the questionnaire. Generally, there is a quick turnaround of responses. Cost factors include printing costs and data entry time.
Written Questionnaires Completed via U.S. Post	With a high response rate, the data will represent all GED attainers (who may be in varied regions or locations). Data collection may require several attempts to achieve a high response rate. Cost factors include printing costs, mailing costs, including additional costs for multiple attempts, and data entry time.
Written Questionnaires Completed via Email or Web-based Survey	With a high response rate, the data will represent all GED attainers (who may be in varied regions or locations). Email or Web-based surveys tend to be relatively fast with regard to distribution and collection. Data collection may require several attempts to achieve a high response rate. There are several technological issues to consider. These include: Web-based surveys provide data in Excel format. This can reduce the amount of time necessary for data entry. GED attainers must have access to and be able to use email and the Internet. Grantees must be able to implement an Internet-based tool. Spam filters may prevent receipt of email/tool and return of data. Cost factors include Internet and email service for grantees, costs of obtaining and hosting a Web-survey service, and data entry time, if using email.
Telephone interviews	With a high response rate, the data will represent all GED attainers (who may be in varied regions or locations). The interviewer can clarify questions or responses if necessary. Telephone numbers must exist and be current to successfully collect data. Caller ID and use of cell phones as primary phones may create difficulties in finding and accessing participants. GED attainers may not answer a call with an unknown caller ID or may change cell phones or cell phone numbers, making older numbers obsolete. Interviews can incur a lot of time related to making individual contacts and conducting interview, especially for grantees with large numbers of participants. Multiple attempts may be necessary to achieve a high response rate. Responses can be directly entered into a database by interviewer, which will reduce the amount of time necessary for data entry. Cost factors include telephone costs, including additional costs for multiple attempts, and interviewer training and data collection time.

Table 1: Considerations of OME HEP Data Collection Methods (continued)

Data Collection Method	Considerations
Face-to-face interviews	If face-to-face interviews are conducted locally only, the data will not include responses from GED attainers in other regions or states. The interviewer can clarify questions or responses if necessary. Interviews can incur a lot of time related to making individual contacts and conducting interviews, especially for grantees with large numbers of participants. Responses can be directly entered into a database by interviewer, which will reduce the amount of time necessary for data entry. Cost factors include local travel costs and costs related to interviewer training and data collection time.
Data matching	Data are highly reliable if grantee ensures that matched data fields provide the precise data needed. This technique requires cooperation of other agencies and the informed (written) consent of participants to share information. Grantees must consult with legal authorities to ensure the system is in compliance with all laws and regulations.9 Data matching requires common, unique identifier between databases. If these do not exist they will need to be developed. In addition, the grantee will need to develop and provide training on the process for data matching. A time lag may exist between reporting deadline and availability of matching data.10 Not all GED attainers may appear in multiple agencies or systems. Thus, data matching may not provide data on all GED attainers. Cost factors include personnel costs related to managing and conducting data matching, and data entry/management time.

We recommend grantees use a combination of methods to contact GED attainers. Some grantees may elect to conduct a written questionnaire and a telephone interview. In other instances, grantees also may use data matching with other agencies to confirm a GED attainer’s post-GED placement status. In these instances, each cooperating agency will share information about the participant, usually via a unique identifier. For example, a grantee may obtain postsecondary education placement information from a local college. Data matching also can be used to supplement telephone contacts or questionnaires, particularly if contacts are low.

Regardless of the data collection method(s) used, the grantee must ensure confidentiality and protect participant identity.

Implementing the data collection

There are several factors to consider when implementing the data collection regardless of whether the grantee selects telephone contact, written questionnaire, data matching, or some combination of methods to determine post-GED placement status. These factors are:

• the timing of the data collection;

• personnel needs; and

• protocol and training.

The grantee also will have to consider other factors that are specific to a data collection approach. These factors are discussed below.

The timing of the data collection

The deadline for completing all follow-up contacts will be set by the project director to coincide with the OME HEP customized APR reporting deadline of December 31^st.

The timing of follow-up efforts will vary across projects according to the timing of GED attainment and post-GED placement. In general, it is presumed that GED attainers will achieve their post-GED placement within 4 weeks of attaining the GED. This, however, may not be the case for every participant, and in some cases, ongoing contact with GED attainers may be necessary. We recommend that grantees start attempting follow-up with their GED attainers as soon as possible but no later than 4 weeks after the attainer leaves the project. Once the grantee has achieved follow-up for a GED attainer and the attainer has achieved a desired post-GED placement (i.e., enrolled in postsecondary education, upgraded employment, or joined the military) the GED attainer does not need to be contacted for this information again.¹¹ For those GED attainers who could not be contacted during the first attempt or did not have a desired post-GED placement at the time of the first contact, we recommend the grantee continue to attempt follow-up to collect their placement data, up until the project’s annual internal reporting deadlines.

Personnel needs

Project directors will need to ensure there are staff and time to fully implement data collection. Depending on the number of GED attainers who are exiting the project each project period, there may be a large amount of data to collect. The project director needs to make sure there is a sufficient number of staff to conduct the volume of follow up. Additionally, the burden of conducting the follow-up most likely will vary over the course of a project period. For example, the most intensive time should surround preparing for and implementing the first follow-up. It is recommended that the project include the personnel and expenses necessary for follow-up in its annual budgets and scope of work.

In some instances and when project resources permit, the grantee may want to consider contracting with a third party to conduct the follow-up and thus alleviate the burden on project staff.

Protocol and training

The training protocol should contain all the steps necessary to collect valid and reliable data.

Project directors should ensure that a protocol for data collection, whether by telephone contact, written questionnaire or data matching, be developed and sufficient training provided to all personnel who will collect, enter, and manage data. The protocol and training should, at a minimum, include all the steps necessary to collect valid and reliable data, either from the GED attainers themselves or through data matching. This means, for example, assuring that all data collectors ask the same questions in the same way. Sample questions for an interview or written questionnaire are provided later in the document.

The protocol and training should include any steps necessary for tracking which GED attainers have been contacted or matched and the results of the attempted follow-up. The Program Office requires grantees to maintain some form of supporting documentation if a GED attainer achieves a desired post-GED placement. Documentation can include any documents collected from supporting sources (such as pay stubs from a GED attainer’s employer or registration forms from an attainer’s postsecondary institution) or case notes indicating the date and method of follow-up with the GED attainer (e.g., telephone interview, questionnaire, etc.) during which post-GED status was ascertained. Training also should include steps on how to protect participant identity.

Additional factors specific to telephone contacts and written questionnaires

Constructing the instrument. In instances of telephone contact, many grantees will use one or more staff to conduct follow-up with individual GED attainers. When this is the case, or when grantees decide to use a written questionnaire, it is important that a uniform instrument be in place. When constructing an instrument, grantees should keep in mind:

• The instrument should be as short and simple as possible and should not be worded or presented such that the questions bias or influence responses. A short follow-up protocol is provided below, as an example. In this sample protocol, a GED attainer may answer “yes” to more than one question.

Sample protocol wording: “Since the time you earned your GED …

• “Have you entered postsecondary education?”

• “Have you moved into a job or position that is both full time and salaried?”

• “Have you moved into a job with increased or better benefits, compared to the position you had before?”

• “Have you received a position upgrade or promotion with the same employer?”

• “Have you moved into a job with a predefined career ladder or opportunities for promotion, where you weren’t in this type of position before?”

• “Have you moved into a job with higher hourly wages, compared to the position you had before?”

• “Have you joined the military?”

• The telephone contact or written questionnaire should be conducted in the primary language spoken by the GED attainer, if possible.

• Contact with the GED attainer should be personalized as much as possible by using the GED attainer’s name.

• The instrument should include language letting GED attainers know how important their participation is to the follow-up effort and ensuring them that their data will be kept confidential and their identities will not be used in reporting.

Additional staff training. As noted, the project director should provide training for the staff who will be contacting the attainers. In general, this training should encompass all the steps necessary to collect valid and reliable data from GED attainers as well as any steps necessary for tracking which GED attainers have been contacted and the results of the attempted contacts. Additional common areas to cover in the telephone contact training include:

• How to introduce yourself and solicit participation;

• How to ask the questions; and

• How to respond to questions that GED attainers may have.

Training items that are common to both telephone contact and written questionnaires include:

• How to record responses;

• How to track contact/survey attempts; and

• How to follow-up on unsuccessful contact/survey attempts.¹²

Additional factors specific to data matching

Securing agreements. Agreements such as Memoranda of Understanding are needed between cooperating agencies as to the nature of the data that can and will be shared. Grantees need to ensure that the data collected by the cooperating agencies match those needed for GPRA reporting.

Data confidentiality. State law may prohibit the use of SSNs for data matching or the sharing of data altogether. In other cases, such sharing may be possible with the informed written consent of the participants. In these cases, a process for verifying SSNs will be necessary. If SSNs cannot be used, then an agreement regarding the standardized construction of unique identifiers will be needed. Regardless of the approach, establishing data matching will require consultation with legal authorities to ensure the system is in compliance with all laws and regulations.

Infrastructure development. Grantees will have to construct a database of participant information to facilitate the sharing of data.

Exhibit 1 provides a summary of data collection tips.

Exhibit 1. Summary of data collection tips

• Alert GED attainers that they will be contacted to report on their post-GED placement status.

• Maintain multiple addresses and contact numbers for participants.

• Have a sufficient number of staff to support the data collection.

• Train all data collectors.

• Start follow-up attempts as soon as possible but no later than 4 weeks after attainer leaves the project.

• Conduct multiple (at least three) contact attempts with participants. This can be achieved by sending an introductory letter and reminders by written mail, if addresses are available, or by attempting multiple telephone contacts when necessary.

• Personalize the contact as much as possible by using the participant’s name.

• Have a standardized instrument, and keep questions as short and simple to understand as possible.

• Let participants know how valuable (or helpful) their information will be to the project.

• Ensure participants that their data will be kept confidential and their identities will not be used in reporting.

• Use multiple data collection methods to achieve high response rates.

• Explore data matching with local agencies.

• Communicate findings of follow-up to GED attainers.

Entering and managing data

Procedures for implementing follow-up should also include steps for handling the data that are received. The primary step is data entry into a data management system such as a spreadsheet or database.

We recommend implementing one spreadsheet or database per reporting year to track the number of participants in the project, the number of GED attainers during the reporting year, the number of GED attainers for whom follow-up or data matching was attempted, the number of GED attainers for whom follow-up was achieved, and the number of GED attainers who achieved desired post-GED placements during the reporting year. Figure 1 provides an example of how a spreadsheet might be constructed to capture these data. Such a spreadsheet is for the grantee’s use only. Do not submit this spreadsheet to the OME HEP Program Office. (Figure 1 is divided into two sections due to space restrictions. As a spreadsheet file, all of figure 1 can be incorporated into one spreadsheet page).

As shown in figure 1, we recommend the grantee track:

1. Participant’s last name and first name

2. Whether or not the participant attained a GED

3. If the participant attained a GED, the date the participant attained the GED

4. Date of 1^st follow-up attempt and all subsequent attempts

5. Result (e.g., successful contact) of 1^st follow-up attempt and all subsequent attempts

6. Actual placement or response from GED attainers for whom follow-up was successful.

7. Source of documentation of a GED attainer’s post-GED placement (e.g., copy of pay stub, copy of college registration, etc.).

Using figure 1 as an example, the grantee would be able to track:

1. Number of GED attainers,

2. Number of GED attainers for whom there was successful follow-up, and

3. Placement status of GED attainers for whom there was successful follow-up.

The information presented in this document is an introduction, but not a comprehensive guide, to conducting surveys. Grantees seeking additional information are advised to follow-up with references listed in the appendix.

Figure 1: Sample Follow-Up Spreadsheet Configuration

Section 1

Participant Last Name	Participant First Name	Attained GED (yes/no)	Date attained GED	Date of 1st follow-up attempt	1st follow-up result (success-ful or not)	Date of 2nd follow-up attempt	2nd follow-up result (successful or not)	Date of 3rd follow-up attempt	3rd follow-up result (successful or not)	*

*Enter additional columns for additional follow-up attempts

Section 2

Did GED attainer achieve desired placement? (1 if yes, 0 if no)	Actual placement or response: entered post secondary education	Actual placement or repsonse: upgrade employment	Actual placement or reponse: entered the military	Source of documentation of post-GED placement

Appendix

References on Survey Development and Implementation

Converse , J.M and Presser, S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire (Quantitative Applications in the Social Sciences). Thousand Oaks, CA: Sage Publications.

Czaja, D. (2004). Designing Surveys: A Guide to Decisions and Procedures. 2^nd edition. Thousand Oaks, CA: Sage Publications.

Dillman, D. (2006). Mail and Internet Surveys: The Taliored Design Method 2007 Update with New Internet, Visual, and Mixed-Mode Guide. New York, NY: Wiley—Interscience.

Fink, A. (2005). How to Conduct Surveys: A Step-by-Step Guide. 3^rd edition. Thousand Oaks, CA: Sage Publications.

Fowler, F. (1995). Improving Survey Questions: Design and Evaluation. Thousand Oaks, CA: Sage Publications.

Fowler, F. (2008). Survey Research Methods. 4^th edition. Thousand Oaks, CA: Sage Publications.

Groves, R.M, Biemer, P.P., Lyberg, L.E., Massey, J.T., Nicholls II, W.L., Waksberg, J., editors (2001). Telephone Survey Methodology. New York, NY: John Wiley and Sons, Inc.

Groves, R.M, Dillman D.A, Eltinge, J.L., and Little, R.J.A. (2002). Survey Nonresponse. New York, NY: John Wiley and Sons, Inc.

Groves, R.M, Fowler Jr., F.J., Couper, M.P., Lepkowski, J.M., Singer, E. and Tourangeau, R. (2004). Survey Methodology. New York, NY: John Wiley and Sons, Inc.

Harkness, J.A, van de Vijver, F.J.R., and Mohler, P.P. (2002). Cross-Cultural Survey Methods. New York, NY: John Wiley and Sons, Inc.

Lepkowski, J.M., Tucker, C., Brick, J.M., De Leeuw, E.D., Japec, L., Lavrakas, P.J., Link, M.W., and Sangster, R.L. (2007). Advances in Telephone Survey Methodology. New York, NY: John Wiley and Sons, Inc.

Punch, K.F. (2006). Survey Research: The Basics (Essential Resource Books for Social Research). Thousand Oaks, CA: Sage Publications.

Stacy, S.E. and Moyer, K.L. (1982). A Guide to Survey Development: Manual on Writing a Survey. ERIC # ED222538.

1 Information about data collection approaches used by “teacher-related” Program Offices that have not received technical assistance through the DQI contract is forthcoming. DQI is gathering this information as part of a larger effort to develop an inventory of performance measures for ED Program Offices that award grants to provide services to teachers or individuals training to become teachers.

2 In particular, OSERS requested information on other program’s data collection in measuring:

a)  The scholars who graduate, the number and percentage of scholars who teach in high-need schools as defined by --

(1) the Administration’s 2010 ESEA reauthorization proposal or (2) high-need LEA as defined by Race to the Top, “High-need LEA means an LEA (a) that serves not fewer than 10,000 children from families with incomes below the poverty line; or (b) for which not less than 20 percent of the children served by the LEA are from families with incomes below the poverty line.”;

b) The scholars who graduate, the number and percentage of scholars who teach in a school for at least three years; and

c) The scholars who graduate, the number and percentage who receive a positive teacher evaluation.

3 Note that as part of the OMB approval process, the Program Office had to submit for clearance and approval a System of Records Notice.

4 The Program Office collects information on enrollment status since participants may be able to defer service payback with full-time enrollment in a degree-granting program.

5 Note: grantees that expect to have 150 or more GED attainers during a project year may be permitted to select a sample of GED attainers for follow-up to respond to GPRA Measure 2. Grantees interested in sampling must request approval from the OME HEP Program Office before selecting a sample and must follow sampling procedures required by the Program Office.

6www.nrsweb.org

7 Grantees that receive permission to sample should tell all GED attainers that they might be contacted after exiting the program to provide this information.

8 Appendix contains additional references for survey development and implementation.

9In cases where the unique identifier is a Social Security Number (SSN), data sharing may not be permissible under law

10 NRS guidelines indicate the time lag may range from one quarter to one year in length. Grantees need to contact cooperating agencies and institutions to determine lengths of time lags. It is possible that a considerable time lag may prevent data matching from being a viable option for GPRA reporting.

11 The grantee may elect to conduct multiple follow-ups with GED attainers in the hope of collecting data on multiple post-GED placements, but this is not needed for reporting of GPRA Measure 2.

12 References provided in the Appendix contain strategies and advice on following up after unsuccessful attempts.

1