NonSub Change Memo - HPOG 2.0 - 15 mos survey change

DATE: September 11, 2018

TO: Steph Tatham

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

FROM: Nicole Constance and Hilary Forster

Office of Planning, Research, and Evaluation (OPRE)

Administration for Children and Families (ACF)

SUBJECT: Request for Non-Substantive Change to Health Profession Opportunity Grants (HPOG) Second Generation National and Tribal Evaluation (OMB Control Number 0970-0462)

This memo requests approval for non-substantive changes to two information collection efforts approved under OMB Control Number 0970-0462. First, it seeks approval for revisions to shorten and improve the clarity of questions in Instrument 12: Short-term (15-month) Follow-up Survey for the Second Generation Health Profession Opportunity Grants (HPOG 2.0) National Evaluation’s impact study (OMB Control Number 0970-0462 approved on 6/8/2018). These revisions are primarily based on findings from a pretest of the instrument. Second, it requests approval for adding a new question to the Short-term Follow-up Survey (Instrument 12) to improve data quality. Finally, it requests approval for a small increase in burden for the in-person implementation interview guide (Instrument #4), previously approved in June 2017. This increase in burden is necessary in order to expand the focus area site visits to be conducted in support of the HPOG 2.0 National Evaluation’s descriptive study.

Non-Substantive Change Request #1—Short-term (15-month) Follow-up Survey

The Short-term Follow-up Survey captures data on the following outcomes of interest to the HPOG 2.0 impact study:

• Section A: training experiences and employment history from the point of random assignment through the interview date.

• Section B: school experiences, containing a loop for each school attended since random assignment.

• Section C: credential attainment

• Section D: conditions of current or most recent employment.

• Section E: household composition.

• Section F: income and financial well-being.

• Section G: 21^st century and cognitive skills.

• Section H: respondent and secondary contact information.

The survey was approved by OMB under OMB Control # 0970-0462 on June 8, 2018. The study team completed a pretest of the survey with 17 HPOG 2.0 national evaluation participants in July 2018. The instrument was programmed for administration via computer assisted personal interviewing (CAPI). Overall, the questionnaire worked well in CAPI; both the interviewers and respondents understood most of the questions.

Although the interviewer confirmed that the instrument flows well, the study team identified a number of small, non-substantive changes that would improve the clarity of the questions and the quality of the responses. Below, we briefly discuss the recommended changes for which we request approval. The exact changes are indicated in track changes in the accompanying survey. Minor changes to instructions to interviewers or survey programmers are indicated in the file but not discussed here.

This request includes changes to the survey instrument for four primary purposes:

• Revising or dropping questions to reduce the administration time;

• Adding logic checks or follow-up probes to improve data quality;

• Correcting the skip logic (i.e., which question to ask next based on the respondent’s previous responses); and

• Adding introductory or clarifying text to explain the purpose of a question.

We summarize the requested changes by type of change in this memo. The revised survey instrument is attached: Instrument 12 HPOG2.0 Short-Term15-Month Follow-up Survey_REV08132018_Trackchanges.

Changes to Reduce Administration Time

The burden estimate for this survey was 60 minutes. During the pretest the interview length averaged 62.2 minutes across 14 completed interviews.¹

The pretest showed that Section B (school experiences) accounted for nearly one third of the interview duration (averaging over 20 minutes). This section includes the school financing questions, which some respondents had difficulty answering and complained that they were repetitive. In the approved survey, this module is repeated for every school spell.

In an effort to reduce the overall response burden, we propose two changes in Section B:

1. We seek approval to reduce Section B by reducing the number of times respondents with more than one school spell are asked B5-B16 (questions related to school financing and service receipt). Rather than repeating these questions for every school spell, we propose to ask them for up to two spells. This strategy collects information for all spells for respondents with one or two spells, which we believe will be the majority of respondents. For respondents with three or more school spells, we will ask these questions of the school spell with longest period of attendance and then randomly select a second spell to ask about. Information is always collected for the longest spell—for which costs are likely to be largest. The random selection of the second spell will support high quality imputation for the spells not collected.

2. We seek approval to reformat B6a-k (sources of funding for school or living expenses). Originally the question asked if the respondent used each funding source to pay for school and/or living expenses and then probed to determine which type of expense it was used for. We seek approval to ask B6a-k in one loop, only capturing whether or not the respondent used the source to pay for school expenses.

Respondents also felt that Item C1d (job duties for those working in the field they are currently studying) and Items D3 and D4 were repetitive. We agree and seek approval to drop Item C1d from the survey. The same information will be captured in Section D as part of a more comprehensive industry and occupation module.

We seek approval to drop questions F11a-F11i. Upon further review, we believe we can get what we need analytically with F12 only. 

Adding programming quality checks

B1b and B1c: We seek approval to add a probe following these two questions to confirm the date that the respondent stopped taking classes temporarily and the date that the respondent returned.

Correcting skip logic

B11a and B11f: We seek approval to add a skip instruction so that anyone who responds that they do not have children of childcare age in Items A7a or A7b are not asked the subsequent childcare questions.

B18-B24: We seek approval to add skip logic to this module. If the respondent started all school spells before random assignment, we will not ask B18-B24 at all. Otherwise, we will ask B18-B24 of the spell with the earliest start date after random assignment.

C7: We seek approval to skip the questions about work-based training for those who are self-employed.

D1: We seek approval to skip those who report working for a state, local, or federal government agency out of question D2 (industry).

Adding Introductory or Clarifying Text to a Question

A1: throughout Section A we refer to respondents as “working for pay, taking classes, both or neither”. In A1, the question asked about going to school as opposed to taking classes. We seek approval to change “going to school” to “taking classes” in A1 and elsewhere throughout Section A to make Section A consistent.

A7: We seek approval for two changes here. First, we seek approval for changes to the introduction to simplify it. Second, we seek approval to change the language in Items A7a and A7b to remove the reference to “typically” and replace it with “ever” to ensure that any use of childcare is captured in subsequent questions. We believe removing the word “typically” will improve data quality.

B1a: We seek approval to add clarifying text indicating that a respondent should exclude summer breaks when answering this question.

C1e: We seek approval to change the descriptions of the job duties to better relate to HPOG 2.0 jobs.

C3: We seek approval for minor modifications to the introduction to C3 and C3a-c to help clarify the purpose of the question for respondents.

E6, E7 and E8: It is possible that a respondent’s gender could be wrong in the sample file. We therefore seek approval to add text for interviewers to use if they learn the gender in our files is incorrect.

E7: We seek approval to streamline this question to simply ask male respondents if they have fathered a child since RAD.

F1j: We seek approval to add examples of in-kind sources of income (diapers, groceries).

F14: We recognize that younger respondents may be covered under a parent’s insurance so we seek approval to add that to the question text.

G2 and G3: We seek approval to add introductory text to help the respondent transition between everyday use of computer, reading, and math skills.

Modifications to Improve Data Quality

A8_1: If a school name provided in A8 does not match one on the IPEDS list, we attempt to code the school based on information about the credential that the respondent is pursuing. We seek approval to add a follow-up question to capture street name, cross-street, and city if school name was not found in IPEDS list. This will aid our ability to code the type of school and improve data quality.

A11 (for job spells): The approved instrument only asks whether a job was full-time or part-time. This was an error in our earlier submission. We seek approval to change the format of A11 (for job spells only) to ask how many hours per week the respondent worked at each job. (Note: The corresponding question for school spells already captures actual hours of school attendance.)

B14: In the approved instrument, this question captured only a categorical report of hours spent in one-on-one training. This was an error in our earlier submission. We seek approval to format B14 the same way as B15; i.e., to first attempt to collect actual hours spent in one-on-one training. If the respondent does not know the answer, then we would collect the categorical response in B14a.

Expected Benefits

We expect these requested changes to improve the overall quality of the instrument design and flow in several ways. First, these changes will reduce the current length of the interview to the expected 60 minutes. Second, the changes will improve the instrument usability and the quality of the data collected. Clearly worded questions improve the interviewer’s ability to administer the questions and the respondent’s ability to respond to them. Third, improving the skip logic ensures that questions are not asked of a participant unnecessarily. Finally, these changes are also expected to moderately reduce the respondent burden. Reductions in burden are projected to be largest for the key group for whom the pretest survey was well over the target 60 minutes—those with three or more school spells.

Non-Substantive Change Request #2: Additional Question for Short-term (15-month) Follow-up Survey

We identified a critical data item that was omitted from the originally approved instrument:

D7-D7c: We inadvertently dropped from the original submission a question to capture the respondent’s earnings at the current or most recent job. We seek approval to add D7-D7a, which will capture respondent earnings. D7b-D7c capture the same data but for respondents that are self-employed. Respondents will answer either D7/D7a or D7b/D7c, depending on whether or not they are self-employed.

The requested additions are include in the attached revised: Instrument 12 HPOG2.0 Short-Term15-Month Follow-up Survey_ REV08132018_Trackchanges

Expected Benefits

These additional questions will add critical data items related to current or recent earnings that were inadvertently omitted from the first draft of the survey. We believe that the burden associated with the addition of these two questions is small and will be offset by the reduction in burden from the other changes we propose.

Non-substantive Change Request #3: Expanded Effort for Site Visits

OMB approved Instrument 4: In-Person Implementation Interview Guide in June 2017, under OMB Control No. 0970-0462. This instrument was approved for administration during site visits to two programs for each of five focus areas, for a total of 10 site visits and 100 completed interviews. The Department seeks approval for a small increase in burden in order to increase the number of site visits and corresponding interviews. Under the expanded effort, the study team will visit three programs for each of the five focus areas. The expanded effort will include a total of 15 site visits and 177 completed interviews. No changes to Instrument #4 are required.

Expected Benefits

The expanded effort will support greater breadth and depth of data collection by visiting one additional program per focus area. The additional site visits will also allow us to include: 1) employer interviews for the employer engagement and career pathways focus areas and 2) partner interviews for the sustainability focus area—neither of which are possible under the original approval. The expansion to include these respondents is of particular interest to both the program and evaluation teams at ACF.

The additional burden resulting from the expanded effort is very small relative to the overall burden for the HPOG 2.0 National Evaluation. The annual burden associated with these extra site visit interviews is 39 hours annually. The total annual burden for the evaluation over the next three years is 11,044 hours—the additional 39 annual hours equate to just over one third of one percent (0.35 percent) of the total annual burden.

1 Three cases were excluded from the timings analysis because they were completed over the course of multiple sessions, by respondent request. Because these cases broke off in the middle of a section, the system was unable to calculate the length of the section accurately. Informal timings for these three cases were similar to or slightly longer than for the other cases for which there was no break off and for which formal timings were available.

5