Justification

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: March 3, 2015

TO: Shelly Martinez, OMB

FROM: Sharon Boivin, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: Employment and Earnings Survey – Second Round of Cognitive Testing (OMB# 1850-0803 v.130)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, pilot tests, focus groups, and cognitive interviews. This request is to conduct a second round of cognitive interviews to aid in development of employment and earnings items for the National Household Education Survey (NHES) program. These items will also inform the U.S. Department of Education’s development of a new gainful employment survey that will be used by postsecondary programs to survey graduates about their earnings. The first round of cognitive interviews for this development effort and recruitment for the second round were approved separately in January 2015 (#1850-0803 v.123 and v.125, respectively). The data collection for this study is being carried out for NCES by RTI International (contract # ED-IES-13-C-0070).

As background, the NHES collects earnings information as part of the Adult Training and Education Survey. This survey is designed to collect nationally-representative data on the attainment of non-degree credentials and the labor market characteristics of individuals with such credentials. As part of its submission for OMB clearance for the full scale 2016 NHES, NCES will propose a small pilot test to compare collecting point estimates versus categories for respondent earnings. If it is feasible to collect point estimates, NCES can derive variables reflecting poverty status to include on public use files, providing licensed analysts will have more sensitivity for modeling the relationship between credential attainment and labor market outcomes. The cognitive interviews described in this submittal will test the proposed survey item language. For ease of reference, the items proposed here for testing are referred to as the Employment and Earnings Survey.

The proposed cognitive interviews will also inform the gainful employment survey, which will be used by institutions to appeal program-level debt-to-earnings ratios calculated by the U.S. Department of Education (ED) under the gainful employment regulations (34 CFR Parts 600 and 668). The earnings component of the debt-to-earnings ratio (D/E ratio) is provided by the Social Security Administration, but institutions may calculate an alternative earnings measure by administering a survey to program graduates. Under the regulations, NCES is responsible for developing the survey and the technical standards to which programs must adhere in its administration. Institutions that choose to submit alternate earnings appeal information will survey all students from programs who graduated during the same period that ED used to calculate the D/E ratios, or a comparable period as defined in 668.406(b)(3) of the regulations. The survey will provide an additional source of earnings data for ED to consider before deciding on final D/E ratios for programs subject to the gainful employment regulations. Programs with final D/E ratios that fail to meet the minimum threshold may face sanctions, including the possible loss of Title IV (federal financial aid) program funds.

Two rounds of cognitive interviews will refine the Employment and Earnings Survey questions in order to maximize the quality of data collected and provide information on issues with important implications for the survey design. The initial survey and plans for its testing were developed with input from a Technical Review Panel (TRP) that met on December 2, 2014 (membership of the panel is provided in Attachment I). The results of the first round of testing, which was conducted in January and February 2015, were shared with a panel convened by the National Institute of Statistical Sciences (NISS) on February 27, 2015 (membership of the NISS panel is included in Attachment II, and a copy of the report from the first round of testing is included as Attachment VI). The most significant recommendation from the first round of testing was to combine all income questions into a single multi-part question in order to help respondents correctly distinguish sources of income without double counting. Neither the time burden imposed by the survey nor the sensitivity of the items was reported as a concern by participants. The most significant recommendation from the NISS panel was to revamp the questions to help respondents understand the need to report income from ALL sources and to help them recall their earnings information from two years prior to data collection. To improve recall, the revised survey asks respondents to focus first on the job they held the longest, and then on all other jobs. This approach, recommended by panel member Katharine Abraham and agreed to by the other panel members, is designed to help respondents think about their actual employment situation in 2013. The goal is to elicit total earnings, not to count earnings for the longest job separately.

This request is to conduct the second round of cognitive testing of the Employment and Earnings Survey. In addition to the aforementioned Attachemnts I, II, and IV, this submission includes the informed consent form for cognitive interview participants (Attachment III), the questionnaire to be tested in the cognitive interviews (Attachment V), and the protocol for conducting the cognitive interviews (Attachment VI).

Design and Context

Like the first round, the second round of cognitive testing of the Employment and Earnings Survey will be conducted for NCES by its contractor, RTI International (RTI), in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago, IL. Participants will be recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. RTI will recruit sufficient numbers of graduates to ensure 15 completed interviews. The recruitment material and the screener for participants were already approved on January 30, 2015 (#1850-0803 v.125).

The desired sample will include a mix of respondents who are (1) recipients of a degree or certificate from any level of private for-profit institutions and (2) recipients of a certificate, associates degree, or other vocational credential at public and private not-for-profit 2-year and less-than-2-year institutions. To help inform gainful employment regulations, the sample will overrepresent graduates from private for-profit less-than-2-year and 2-year institutions. Such schools had higher than average metric failure rates. Table 1 provides a distribution of the desired minimum number of interview participants by control and level of institution.

Table 1: Desired minimum number of respondents, by institution control and level

Institution control	Institution level	Credential type(s)	Minimum number of respondents
Private for-profit	Less-than-2-year	Vocational certificates	4
	2-year	Vocational certificates and associates	4
	4-year	All	2
Private not-for-profit	Less-than-2-year and 2-year	Vocational certificates and associates	2
Public	Less-than-2-year and 2-year	Vocational certificates and associates	3

As in the first round, cognitive testing will involve one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer additional questions about the items they just answered. The survey is being developed and tested as a paper-and-pencil instrument. Each testing session will last about 40 minutes. In order to encourage participation in this study, potential respondents will be offered a $40 incentive for their participation. Arrangements will be made to allow NCES staff to observe the testing and a recording of each interview will be available to NCES for review.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student’s name in any way. Participants will sign an informed consent form (Attachment III) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study and destroyed after the final report is submitted.

Schedule for Development of the Employment and Earnings Survey

This second round of interviews is scheduled to take place in mid March 2015. The final draft of the survey will be used in a pilot test with approximately 3,400 program graduates beginning in April 2015. All survey item development activities need to be completed by the end of June 2015 for inclusion in the clearance request for the 2016 collection of NHES and to inform the gainful employment regulations that will be posted in the Federal Register by July 1, 2015. The request for the pilot test will be submitted to OMB as soon as all required materials become available.

Estimate of Respondent Burden

To yield 15 completed interviews, we anticipate screening up to 60 individuals for eligibility to achieve the desired distribution of program graduates. The recruitment and screening process, which has been approved and underway, was estimated to take on average 5 minutes per person.

Table 2: Estimate respondent burden for recruitment for second round of cognitive interviews

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening*	60	60	5	5
Cognitive interview	15	15	40	10
Study Total	15	15		10

*Burden for screening was approved in January 2015, and thus is not counted in the total burden requested in this submission.

Cost to Federal Government

The cost to the federal government for developing and cognitive and pilot testing the survey will be $555,570, which includes contractor staff time, incentives, and project materials.

Attachment I

Employment and Earnings Survey (EES) Technical Review Panel Members

Geri Anderson

Special Asssistant to the President for External Affairs

Aims Community College

5401 West 20th Street

Greeley, CO 80632

Phone: (303)503-4074

Email: geri.anderson@aims.edu

Angela Bell

Senior Executive Director of Research and Policy Analysis

University System of Georgia

270 Washington Street SW

Atlanta, GA 30334

Phone: (404)962-3069

Email: angela.bell@usg.edu

Jennifer Blum

Senior Vice Present, External Relations & Public Policy

Laureate Education, Incorporaated

1500 K Street NW, Suite 250

Washington, DC 20005

Phone: (202)393-1303

Email: jennifer.blum@laureate.net

Kathy Booth

Senior Research Associate

WestEd

300 Lakeside Drive, 25th Floor

Oakland, CA 94612

Phone: (510)302-4208

Email: kbooth@wested.org

Patrick Crane

Project Manager

West Virginia Community & Technical College System

1018 Kanawha Boulevard East, Suite 700

Charleston, WV 25301

Phone: (304)558-0087

Email: crane@wvctcs.org

Mark DeFusco

Senior Research Associate

University of Southern California

Rossier School of Education

Waite Phillips Hall WPH 701B

Los Angeles, CA 90089-4037

Phone: (213)740-7218

Email: mbd_001@usc.edu

Christine Fuglestad

Director of Government Affairs

Capella University

225 South 6th Street, 9th Floor

5019 Wentworth Avenue

Minneapolis, MN 55419

Phone: (612)977-4524

Email: christine.fuglestad@capella.edu

Alfred Gottschalck

Chief, Labor Force Statistics Branch

US Census Bureau

4600 Silver Hill Road

Washington, DC 20233

Phone: (301)763-5883

Email: alfred.o.gottschalck@census.gov

KC Greaney

Director, Office of Institutional Research

Santa Rosa Junior College

680 Sonoma Mountain Parkway

Richard Call Bldg., Annex

Petaluma, CA 94954

Phone: (707)778-4188

Email: kgreaney@santarosa.edu

Stephen Haworth

Senior Manager, Reporting & Policy Research

Devry Education Group

3005 Highland Parkway

Downers Grove, IL 60515

Phone: (630)353-1434

Email: shaworth@devrygroup.com

G. Scott Jenkins

Vice Provost for Academic Affairs and Undergraduate Programs

North Carolina A&T State University

1601 East Market Street

Dowdy Building 318

Greensboro, NC 27411

Phone: (336)285-3039

Email: gsjenkin@ncat.edu

Anthony Jones

Graduate Faculty

Appalachian State University

151 College Street, Suite 217-B

Boone, NC 28608

Phone: (828)262-2287

Email: jonesap1@appstate.edu

John Kolotos

Policy Analyst

U.S. Department of Education

1990 K Street NW

Washington, DC 20202

Phone: (202)502-7762

Email: john.kolotos@ed.gov

Tod Massa

Director, Policy Research and Data Warehousing

State Council of Higher Education for Virginia

101 North 14th Street

Richmond, VA 23219

Phone: (804)225-3147

Email: todmassa@schev.edu

Heather McKay

Director, Education and Employment Research Center

Rutgers University

94 Rockafeller Road

Piscataway, NJ 08854-8054

Phone: (848)445-4735

Email: hmckay@rci.rutgers.edu

Charles Nelson

Assistant Division Chief, Economic Characteristics

Census Bureau

4600 Silver Hill Road

Washington, DC 20233

Phone: (301)763-3183

Email: charles.t.nelson@census.gov

Kent Phillipe (unable to attend)

Associate Vice President, Research & Student Success

American Association of Community Colleges

One Dupont Circle NW

Suite 410

Washington, DC 20036

Phone: (202)416-4505

Email: kphillippe@aacc.nche.edu

Anne Polivka

Supervisory Research Economist, Employment Research Chief

Bureau of Labor Statistics

2 Massachusetts Avenue NE, Suite 4945

Washington, DC 20212

Phone: (202)691-7395

Email: polivka.anne@bls.gov

Casey Sacks

Manager

Colorado Community College System

9101 East Lowry Boulevard

Denver, CO 80230

Phone: (720)858-2841

Email: casey.sacks@cccs.edu

Rajat Shah

Senior Vice President, Student Financial Services

Lincoln Technical Institute

200 Executive Drive, Suite 340

West Orange, NJ 7052

Phone: (973)766-9099

Email: rshah@lincolnedu.com

Christine Tracy

Director of Research

Association for Private Sector Colleges and Universities

1100 Connecticut Avenue NW

Suite 900

Washington, DC 20036

Phone: (202)336-6804

Email: chris.tracy@apscu.org

Christina Whitfield

Vice Chancellor

Kentucky Community & Technical College System

300 North Main Street

Versailles, KY 40383

Phone: (859)256-3184

Email: christina.whitfield@kctcs.edu

Paul Umbach

Professor, Higher Education and Educational Evaluation and Policy Analysis

Department of Leadership, Policy, and Adult and Higher Education

North Carolina State University

300 Poe Hall, Box 7801

Raleigh, NC 27695

Phone: (919)515-9366

Email: paul_umbach@ncsu.edu

National Center for Education Statistics

Sharon Boivin

Mathematical Statistician

U.S. Department of Education, NCES

1990 K Street NW, Room 9052

Washington, DC 20006

Phone: (202)502-7627

Email: sharon.boivin@ed.gov

Chris Chapman

Acting Associate Commissioner, Sample Surveys

U.S. Department of Education, NCES

1990 K Street NW, Room 9042

Washington, DC 20006

Phone: (202)502-7414

Email: chris.chapman@ed.gov

Tracy Hunt-White

Education Statistician

U.S. Department of Education, NCES

1990 K Street NW, Room 9018

Washington, DC 20006

Phone: (202)502-7438

Email: tracy.hunt-white@ed.gov

Marilyn Seastrom

Chief Statistician

U.S. Department of Education, NCES

1990 K Street NW, Room 9040

Washington, DC 20006

Phone: (202)502-7303

Email: marilyn.seastrom@ed.gov

Sean Simone

Statistician

U.S. Department of Education, NCES

1990 K Street NW, Room 8125

Washington, DC 20006

Phone: (202)502-7367

Email: sean.simone@ed.gov

RTI International

Jeff Franklin

Senior Survey Methodologist

RTI International

3040 East Cornwallis Road

P.O. Box 12194

Research Triangle Park, NC 27709

Phone: (919)485-2614

Email: jwf@rti.org

Alan Karr

Director, Center of Excellence for Complex Data Analysis

RTI International

3040 East Cornwallis Road

P.O. Box 12194

Research Triangle Park, NC 27709

Phone: (919)316-3423

Email: karr@rti.org

T. Austin Lacy

Education Research Analyst

RTI International

3040 East Cornwallis Road

P.O. Box 12194

Research Triangle Park, NC 27709

Phone: (919)990-8386

Email: tlacy@rti.org

John Riccobono

Vice President, Education & Workforce Development

RTI International

3040 East Cornwallis Road

P.O. Box 12194

Research Triangle Park, NC 27709

Phone: (919)541-7006

Email: jar@rti.org

Peter Siegel

Senior Statistician

RTI International

3040 East Cornwallis Road

P.O. Box 12194

Research Triangle Park, NC 27709

Phone: (919)541-6348

Email: siegel@rti.org

Erin Velez

Education Research Analyst

RTI International

701 13th Street NW

Suite 750

Washington, DC 20005

Phone: (202)974-7879

Email: evelez@rti.org

Jennifer Wine

Senior Director, Program in Education Survey Design

RTI International

3040 East Cornwallis Road

P.O. Box 12194

Research Triangle Park, NC 27709

Phone: (919)541-6870

Email: jennifer@rti.org

Attachment II

NCES/NISS Gainful Employment Survey Content Panel Members (February 27, 2015 - Washington, DC)

Katherine Abraham

Professor of Economics and Survey

Methodology

University of Maryland

1218 LeFrak Hall

College Park, MD 20742

Email: kabraham@umd.edu

Margaret Ayanian

Business Operations

U.S. Department of Education

Federal Student Aid (FSA)

830 First Street NE

Washington, DC 20202

Email: Margaret.Ayanian@ed.gov

Sharon Boivin

Chair, Interagency Working Group

on Expanded Measures of

Enrollment and Attainment

National Center for Education

Statistics

1990 K Street NW

Washington, DC 20006

Email: Sharon.Boivin@ed.gov

Chris Chapman

Associate Commissioner

National Center for Education

Statistics

1990 K Street NW

Washington, DC 20006

Email: Chris.Chapman@ed.gov

Erin Dunlop Velez

Research Education Analyst

RTI International

3040 East Cornwallis Road,

P.O. Box 12194

Research Triangle Park, NC 27709

Email: evalez@rti.org

Michael Larsen

Associate Professor

George Washington University

Rome Hall

801 22nd Street NW

Washington, DC 20052

Email: mlarsen@bsc.gwu.edu

Brent Madoo

Assistant Director

U.S. Department of Education

Federal Student Aid (FSA)

830 F Street NW

Suite 413

Washington, DC 20202

Email: Brent.Madoo@ed.gov

Isaiah O’Rear

Associate Research Scientist

National Center for Education

Statistics

1990 K Street NW

Washington, DC 20006

Email: Isaiah.Orear@ed.gov

Emilda Rivers

Program Director, Human Resources

Statistics

National Center for Science and

Engineering Statistics

National Science Foundation

4201 Wilson Boulevard

Arlington, VA 22230

Email: erivers@nsf.gov

Marilyn Seastrom

Chief Statistician

National Center for Education

Statistics

1990 K Street NW

Washington DC, 20006

Email: Marilyn.Seastrom@ed.gov

Nell Sedransk

Acting Director

National Institute of Statistical

Sciences

19 T.W. Alexander Drive,

P.O. Box 14006

Research Triangle Park, NC 27709

Email: sedransk@niss.org

Joy Edington

ESSIN Research Analyst/Statistician

National Institute of Statistical

Sciences

1776 Eye Street NW

Washington DC, 20006

Email: jedington@niss.org

Martin Frankel

Professor of Statistics

The City University of New York

Baruch College

One Bernard Baruch Way

(55 Lexington Avenue, at 24th Street)

New York, NY 10010

Email: martin.frankel@baruch.cuny.edu

Alan Karr

Director, Center of Excellence for

Complex Data Analysis

RTI International

3040 East Cornwallis Road,

P.O. Box 12194

Research Triangle Park, NC 27709

Email: karr@rti.org

Saki Kinney

Research Statistician

National Institute of Statistical

Sciences

19 T.W. Alexander Drive,

P.O. Box 14006

Research Triangle Park, NC 27709

Email: saki@niss.org

Tushar Sheth

Special Assistant, Planning,

Evaluation, and Policy Development

U.S. Department of Education

400 Maryland Avenue SW

Washington, DC 20202

Email: Tushar.Sheth@ed.gov

Peter Siegel

Statistician

RTI International

3040 East Cornwallis Road,

P.O. Box 12194

Research Triangle Park, NC 27709

Email: siegel@rti.org

Sean Simone

Statistician

National Center for Education

Statistics

1990 K Street NW

Washington, DC 20006

Email: Sean.Simone@ed.gov

Clyde Tucker

Principal Researcher, Education

Program American Institutes for

Research

1000 Thomas Jefferson Street NW

Washington, DC 20007

Email: nctucker@cox.net

Tina Temples

U.S. Department of Education

Federal Student Aid (FSA)

80 F Street NW

Washington, DC 20202

Email: Tina.Temples@ed.gov

Andrew White

Senior Research Statistician,

Statistical Standards and Data Confidentiality

National Center for Education Statistics

1990 K Street NW

Washington, DC 20006

Email: Andrew.White@ed.gov

Attachment III
Participant Informed Consent

Consent to Participate in Research

Title of Research: Employment and Earnings Survey – Cognitive Interview

Introduction

You are being asked to participate in a cognitive interview to help improve a new survey being developed. The survey items will be used by the U.S. Department of Education as part of the National Household Education Study in 2016 and by institutions of higher education to appeal program-level debt-to-earnings rates as calculated by the Department. Before you decide if you want to take part in this study, please read this Informed Consent form so that you understand what the interview is about and what you will be asked to do. This form also tells you who can participate in the interviews, the risks and benefits of participating, how we will protect your information, and who you can call if you have questions. Please ask the researcher to explain anything you don’t understand before you make your decision.

Purpose

The interview is being conducted on behalf of the National Center for Education Statistics, within the U.S Department of Education, by RTI International, a research organization with headquarters in Research Triangle Park, North Carolina.

Participation will involve answering questions and reviewing a survey for an upcoming study. During the interview, we will ask you to complete a survey questionnaire on paper or on a computer and give us your thoughts on the questions you are being asked. In addition, we will ask you questions as you go in order to measure your understanding and interpretation of the questions in the survey. The information from the interviews will help us finalize development of the questions for the national survey instrument.

Procedures

You are one of 15 individuals participating in these cognitive interviews. The interview will be audio recorded to make sure we don’t miss anything that you say and to help us write a report summarizing the results of the interviews. Upon completion of the written report, the recording will be destroyed. Your name or any information that could identify you personally will not be used in the report that we write.

Study Duration

Your participation in the cognitive interview will take about 40 minutes.

Possible Risks or Discomfort

We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.

| Your Benefits |

There are no direct benefits to you from participating in this study.

| Benefits for Other People |

We hope that these interviews will help us produce quality questions for the national survey being developed with the U.S. Department of Education.

| Token of Appreciation for Participation |

You will receive $40 upon completion of the interview.

Confidentiality

RTI International is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (20 U.S.C., Section 9543). Your participation is voluntary. Your responses may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573).

Future Contacts

We will not contact you in the future.

Your Rights

Your decision to take part in this research study is completely voluntary. You can refuse to answer any question and you can stop participating at any time.

Your Questions

If you have any questions about the study, you may call Dr. Jennifer Wine at RTI International (919-541-6870). If you have any questions about your rights as a study participant, you may call RTI’s Office of Research Protection at 1-866-214-2043 (a toll-free number).

YOU WILL BE GIVEN A COPY OF THIS CONSENT FORM TO KEEP.

Your signature below indicates that you have read the information provided above, have received answers to your questions, and have freely decided to participate in this research. By agreeing to participate in this research, you are not giving up any of your legal rights.

Signature of Participant		Date
Printed Name of Participant

I certify that the nature and purpose, the potential benefits, and possible risks associated with participating in this research have been explained to the above-named individual.

Signature of Person Obtaining Consent		Date
Printed Name of Person Obtaining Consent

30