Supporting Statement B
Bureau of Health Workforce Nurse Corps
Supplemental Funding Evaluation
OMB Control No. 0915-XXXX-New
The Bureau of Health Workforce Nurse Corps Programs Evaluation does not employ any statistical methods for selecting respondents to the survey or in-depth interviews. The decision not to use statistical methods to select survey respondents was based on two reasons. First, survey data are intended to complement already-available administrative data in a manner that maximizes the utility of administrative data enabling the most comprehensive evaluation possible. Second, the diversity of the programs and the possible combination of types of sites and individuals necessitates data collection from all eligible respondents to ensure a complete understanding of program impacts.
Potential respondents for the Nurse Corps Participant and Alumni Survey will include the census of current Loan Repayment Program and Scholarship Program participants as of the time of the survey, and alumni who completed their service obligation between 2017 and 2023. Excluding alumni who completed their service before 2017 will help improve data quality as those who completed service prior to 2017 may have difficulty accurately recalling their Nurse Corps experience. An estimated 7,303 individuals are eligible to participate in the survey: 5,082 Loan Repayment Program-Clinician participants and alumni, 804 Loan Repayment Program-Nurse Faculty participants and alumni, and 1,416 Scholarship Program participants and alumni. The target response rate for the survey is 70 percent.
The in-depth interviews will be conducted with 57 participants and alumni representing the range of respondent groups: 18 in-depth interviews will be conducted with Loan Repayment Program-Clinician participants and alumni, 18 in-depth interviews will be conducted with Loan Repayment Program-Nurse Faculty participants and alumni, and 21 in-depth interviews will be conducted with Scholarship Program participants (both in school and completing their service obligation) and alumni. Our primary recruitment approach is the inclusion of a survey question asking respondents if they would be willing to participate in an in-depth interview. If needed, we will recruit directly from the list of program participants and alumni via email. We will aim to maximize diversity within each program in terms of respondent and site characteristics, but we do not propose any statistical methods for selecting participants given the small, targeted number.
We have designed data collection procedures to support a high response rate, reduce burden to respondents, and promote accuracy of the data. The procedures for the collection of information are described below.
Survey Procedures: HRSA will provide Westat with participant and alumni contact lists for both the Loan Repayment Program and Scholarship Program, which will function as the frame from which we will identify the census of current participants and eligible alumni and acquire their contact information (name, email address, mailing address, and phone number).
Initial survey administration will be via web survey, as HRSA is able to provide email addresses that are expected to be reasonably up-to-date and this population is likely to be comfortable with using the web. The web survey provides a flexible opportunity for respondents to complete the survey, which is important for a population that is likely to have busy schedules and many competing demands for their attention. Including mailed notifications to nonrespondents can help reach and engage individuals who may have inaccurate email addresses or who do not regularly check their email. Additionally, telephone follow-up to nonrespondents allows individuals to complete the survey by phone. Table B-1 provides an overview of the survey fielding approach.
Table B-1: Survey fielding process
Protocol stage |
Mode and Steward |
Description |
Timing |
Survey prenotification |
Email - HRSA Bureau of Health Workforce Management Information System Solution notification - HRSA |
Initial email from HRSA through the Nurse Corps listserv and as an announcement in the Bureau of Health Workforce Management Information System Solution to promote the survey |
Survey launch |
First survey contact |
Email – Westat; Postal mail - Westat |
Email from Westat along with a mailed letter that provides information about the survey, including survey QR code/URL and access code |
Within 24 hours from survey launch |
Second survey contact |
Email - Westat |
Email reminder to survey nonrespondents with survey URL and access code |
One week after survey launch |
Third survey contact |
Postal mail - Westat |
Mailed reminder/thank-you to survey nonrespondents with survey QR code/URL and access code |
Two weeks after survey launch |
Fourth survey contact |
Email - Westat |
Final email reminder about the web survey; notice of incentive ($5 Amazon electronic credit) for completion of the web survey |
Four weeks after survey launch |
Telephone follow-up |
Computer-assisted telephone interviewing - Westat |
Westat telephone interview staff will attempt to complete the survey via Computer-assisted telephone interviewing with nonrespondents; each nonrespondent will receive up to seven telephone contacts |
Eight weeks after survey launch |
The first survey contact will occur within a day of completing prenotifications. We will send via email and postal mail a notification letter that includes HRSA, Bureau of Health Workforce, or Nurse Corps branding; information about the survey; and guidance on how to access the web survey (QR code/URL and access code). This first touchpoint containing survey access information must be expertly designed to maximize cooperation. At the same time, we will assure respondents that participation is voluntary. We will also inform respondents that their survey responses will be private, all analyses will be done at the aggregate, not individual level, and their responses will not affect their Nurse Corps funding in any way.
We will follow up with a reminder email to nonrespondents (second survey contact) approximately one week after the launch and will then send a reminder letter via postal mail (third survey contact) another week later. In these communications, we will begin to emphasize that “time is running out to make your voice heard” and restate the value of participation and our promise of confidentiality. Approximately four weeks after the launch, we will send the final email reminder (fourth survey contact) to web survey nonrespondents. Over the course of survey fielding, notifications will also appear on the Bureau of Health Workforce Management Information System Solution to remind individuals to complete the survey.
Telephone follow-up for nonrespondents will begin roughly eight weeks after the survey launch. Our telephone interviewers are securely connected to a centralized software system that allots their individual cases for calling and provides only the information that the interviewer needs to conduct an interview, such as the name of the person they are calling and the telephone script. Their work will be monitored closely, and especially effective interviewers may be trained for refusal conversions. We will make up to seven telephone contact attempts at different days and times before closing each nonresponse case. We anticipate needing between 25-35 interviewers to cover the volume of calls over the timeframe we envision (1-2 months). We will also assign an Operations Manager to train and monitor telephone interviewers.
Training for telephone survey interviewers: All new hires and returning interviewers receive general interviewer training, which concentrates on the telephone interviewing conduct and protocols (including consent and contact procedures), telephone etiquette and diction, survey interviewer best practices, management of respondent inquiries, and telephone administration and coding procedures that apply across all of our projects. Success in the general interviewer training program is an important indicator of subsequent high quality data collection work. Thus, only those trainees who successfully master general interviewer training are eligible to participate in specific training related to the data collection instruments in this OMB package.
Project-specific training has three primary goals:
Providing background information about the study so that interviewers can represent the study accurately and professionally to respondents.
Reviewing the survey instrument so that interviewers understand the survey content and can administer the survey with confidence to effectively collect high-quality data, without introducing bias or influencing responses.
Practicing interactions with respondents so that interviewers effectively gain respondent cooperation and maintain respondent engagement.
For the Nurse Corps evaluation, we will implement a three-phase project-specific training process. First, trainees complete self-paced modules via our learning management system to become familiar with the study background, purpose, nature of the questionnaire, and data stewardship. Trainees are tested for mastery of the content and only those who pass move on to the next training phase. In the second phase, trainees attend a live, web-based, trainer-led session that focuses on gaining cooperation, addressing respondent questions and concerns, and the unique details of the questionnaire. The trainer-led session provides opportunities for trainees to read the survey script while being evaluated by Westat managers and ask questions. The third phase consists of role play exercises where one trainee administers the survey to another trainee over the phone as if conducting a live interview, before reversing roles and repeating the process. During role play sessions, the interviewer playing the respondent follows scripts covering key survey sections and critical training points. Westat supervisors monitor the role plays to evaluate and determine interviewer readiness for live data collection.
Telephone survey quality control: Once data collection begins, staff will monitor instrument and interviewer performance. Additionally, staff will track productivity and use that information to guide adjustments to outreach techniques. Live monitoring of telephone interviewers administering the survey is the most valuable quality control process available in telephone interviewing. Supervisory staff will monitor telephone data collection regularly throughout the entire data collection period. We will conduct a higher percentage of monitoring in the beginning of data collection to ensure training success and address any issues early on. Interviewers who achieve lower cooperation rates will be mentored and coached more frequently to identify any aspects of their interviewing technique that needs improvement. Individuals who deviate from expectations will be quickly coached, and those who are unable to meet expectations after coaching will be released.
Reporting on survey data collection efforts:
Westat will prepare and submit to the COR a biweekly report on the progress of information collection efforts during the survey data collection period. The reporting will include the number of completed surveys, non-response follow-up status, problems encountered, and proposed solutions.
In-depth interview procedures: To maximize recruitment for the in-depth interviews, we will leverage a two-pronged recruitment approach and offer a $75 check as an incentive. As a first approach, we will identify individuals who are interested in participating in an interview through the survey. The final question on the survey will ask respondents if they are interested in participating in a future interview about their Nurse Corps experiences, and if so, to provide contact information for follow-up. We will include information about the in-depth interviews (purpose, length, incentive) as part of the survey question. If needed, we will conduct email outreach to samples of eligible Nurse Corps participants and alumni. In the outreach email, interested individuals will be prompted to respond to a few screening questions and to provide contact information for follow-up. Table B-2 provides an overview of the in-depth interview process.
Table B-2: In-depth Interview Process
Protocol stage |
Outreach Mode and Steward |
Description |
Timing |
Identify interested respondents |
Survey - Westat |
Use question on survey inviting participants to participate in in-depth interviews to identify the pool of interview candidates |
In-depth interview launch (6 weeks after conclusion of survey data collection) |
Recruiting |
Email (if needed) - Westat |
If recruiting through the survey is insufficient, direct email outreach to Nurse Corps participants and alumni |
2 weeks after In-depth interview launch |
Scheduling |
Email and/or phone - Westat |
Communication to settle on a time for the interview |
On a rolling basis as respondents reply |
Conduct interviews |
Zoom for Government - Westat |
Data collection |
On a rolling basis as respondents are scheduled |
Iterative recruiting and scheduling as needed |
Email and/or phone - Westat |
Communication to recruit/schedule (if needed) |
On a rolling basis as additional respondents are recruited and scheduled |
Participants and alumni who indicate interest in participating in an in-depth interview will be added to a candidate database, which will also contain other relevant information (e.g., program type [i.e., Loan Repayment Program-Clinician, Loan Repayment Program-Nurse Faculty, Scholarship Program], year of award, demographic characteristics, etc.). We will contact eligible individuals by email to schedule the in-depth interviews, aiming to find a time that is convenient for the participant. After scheduling in-depth interviews, we will send participants a confirmation email with the date and details for joining the Zoom for Government session. We will send a reminder by email to each scheduled participant roughly 24 hours in advance.
Each in-depth interview will last no longer than 45 minutes and will be conducted using Zoom for Government. Once the participant has been admitted to the session, the interviewer will introduce themselves and provide background regarding the purpose of the interview. Following a semi-structured discussion guide, the interviewer will remind the participant that their participation in the in-depth interview is voluntary and will assure them that their comments will not be linked with their name in any report or affect their Nurse Corps award in any way. After covering the background and consent information, the interviewer will obtain the participant’s consent to proceed with the interview and to have it recorded (with video turned on or off depending on participants’ preferences). If a participant does not consent to participate in the interview, the interviewer will thank the participant and terminate the session. If a participant consents to participating in the interview, but does not agree to have the discussion recorded, the interviewer will proceed without turning on the recorder and will rely on notetaking to capture the participant’s responses and sentiments.
In-depth interview interviewer training: Each Westat interviewer will be experienced in conducting in-depth interviews and using Zoom for Government, having previously received extensive training. Prior to the start of the in-depth interviews, the Westat in-depth interview lead will provide project-specific training for the interviewers. During the training, the Westat lead will review the purpose of the interviews, provide an orientation to the semi-structured discussion guides, discuss the recruitment and scheduling process, provide a refresher on Zoom for Government features, and answer any questions. To ensure the in-depth interviews run smoothly, Westat interviewers will be assigned a partner with whom they will practice running through a mock interview. Interviewers with less knowledge about the Nurse Corps program will also attend initial sessions conducted by experienced project staff prior to conducting their own interviews. The Westat interviewers will meet regularly during the in-depth interview period to share what they have heard and discuss any concerns or challenges.
In-depth interview quality control: Once in-depth interview data collection begins, staff will monitor instrument and interviewer performance. The in-depth interview task lead will sit in for live monitoring of each interviewer’s first in-depth interview and provide coaching as needed afterwards.
Reporting on in-depth interview status:
Westat will prepare and submit to the COR a biweekly report on the progress of information collection efforts during the in-depth interview data collection period. The reporting will include the number of participants scheduled, follow-up, problems encountered, and proposed solutions.
We aim to achieve a response rate of 70 percent, as specified in the scope of work for this project. We believe this is achievable with a combination of techniques for maximizing response. Our data collection team will perform the following key processes to ensure maximal response rates:
Survey Design: The proposed survey, designed by expert methodologists, offers clear and logical question wording and sequencing, with only relevant content displayed for each respondent. Our web survey platform will make it easy for respondents to participate via smart phone or computer. The web-based platform will also allow respondents to save and continue the survey later, which adds convenience for completing the surveys.
Outreach: We plan a multi-pronged outreach approach for both data collections. Outreach will occur from both HRSA and Westat prior to the survey, and multiple non-response follow-ups by both mail and email will occur. In addition, intensive telephone follow-up is planned for nonrespondents to the web-based survey.
Tracing: For this evaluation, HRSA will furnish contact information for potential respondents. We anticipate that this contact information should be reliable for current participants and assume contact information is accurate for at least 85 percent of the list. However, given that the information is older and/or not well-maintained for program alumni, we expect that more effort will be required to contact these potential respondents. Tracking down current contact information will require contact tracing. Cases that are not reachable at the email address or telephone number(s) in the sample will be sent to Westat’s tracing group for locating updated contact information. Westat offers a variety of tracing services to locate and/or verify contact information for study respondents. These products are built from multiple publicly available sources and have been used with success on many Westat projects. In addition, we utilize tracing staff who are dedicated to this activity and are therefore experienced in finding subjects efficiently through our rigorous and secure processes. Westat will ensure that all information is retained and tracked in a standardized, secure format.
Refusal Avoidance and Conversion: We expect that both HRSA and the Nurse Corps Programs will be familiar names to respondents, which should facilitate cooperation in the survey. However, Westat telephone interviewers are prepared to engage in refusal avoidance techniques to maximize response rates. Refusal avoidance techniques emphasize listening carefully to what the respondent says, neutrally offering appropriate reassurances and encouragement, and not arguing with the respondent. If the interviewer is unable to persuade the respondent to continue the interview using gentle persuasion, the goals are (1) to leave the door open for another attempt, and (2) to learn as much as possible about why the respondent is refusing. Telephone interviewers who are judged to be particularly skillful at refusal avoidance receive special training and are assigned to focus on refusal conversion cases.
Incentives: Respondents to the web-based survey will receive a $5 electronic credit at Amazon as a small token of our appreciation for their time. In-depth interview respondents will receive a check for $75 after completing their interview.
Methods to Address Nonresponse:
Weighting. Although the survey will be fielded to a census of eligible participants, some participants will not respond to the survey request, despite numerous follow-ups. As a result, weights that account and adjust for the nonresponse will be needed for analysis. Weights will be developed to account for survey nonresponse. We will compute nonresponse weights within weighting cells defined by variables that are associated with both key outcome variables and variables that are associated with response propensities and are known for both respondents and nonrespondents. This includes administrative data from the Nurse Corps program participants and secondary data from the linkage of the sampling frame to the administrative data, both of which include demographic, geographic, education, and employment-related data. To create the weighting cells, we will use a classification tree algorithm to determine the variables correlated with survey response. We will post-stratify the nonresponse weights to ensure the weighted respondents correspond to totals from the Nurse Corps program participants included in the original sample by analysis group (population and time cohort). We will provide the final, post-stratified weights on the survey data analysis file and use these weights in all analyses.
Nonresponse Analysis. We will conduct a survey nonresponse bias analysis according to the OMB Standards and Guidelines for Statistical Surveys.1 We will compare the distribution of survey respondents to those of all participants, using variables from the administrative data provided by Bureau of Health Workforce.
4. Tests of Procedures or Methods to Be Undertaken
The survey and in-depth interview instruments were developed in collaboration between Westat and Brandeis University utilizing items from other standardized surveys and reviewed by HRSA workforce subject matter experts. Pilot testing of the survey instrument was conducted with up to nine respondents for each survey question. Pilot test participants were asked to comment on survey length, clarity of instructions and questions, and whether response categories were comprehensive and coherent. Based on feedback from pilot test participants, the survey content was modified as needed to clarify terminology, streamline questions, and refine response options.
Data collection and analysis will be conducted by the Westat evaluation team in partnership with Brandeis University. Contact information for principal staff on the project is listed below.
Health Resources and Services Administration/Bureau of Health Workforce
Stephanie Ziomek, Branch Chief (approval of contract deliverables)
Phone: (301) 443-3359
Email: sziomek@hrsa.gov
Hayden Kepley, Ph.D., Deputy Director
Phone: (301) 443-5339
Email: hkepley@hrsa.gov
Christie Hartman, Ph.D., Statistician
Phone: N/A
Email: chartman@hrsa.gov
Westat
Jennifer Nooney, Ph.D., Project Director (design, collect, analyze data)
Phone: (407) 921-3888
Email: jennifernooney@westat.com
Natalie Teixeira Bailey, MPH, Data Collection Manager (collect data)
Phone: (240) 314-2599
Email: natalieteixeirabailey@westat.com
Brandon Hesgrove, Ph.D., Economist (data analysis)
Phone: (925) 973-9439
Email: brandonhesgrove@westat.com
Jennifer Kali, Statistician (data analysis)
Phone: (301) 738-3588
Email: jenniferkali@westat.com
Brandeis University
Karen Donelan, Sc.D., Principal Investigator (design, analyze data)
Phone: (617) 320-7558
Email: karendonelan@brandeis.edu
1 Office of Management and Budget (OMB). (2006). Standards and Guidelines for Statistical Surveys. Available at: https://www.samhsa.gov/data/sites/default/files/standards_stat_surveys.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | 7420.01: OMB Package. Section A. Introduction |
| Author | MARKOVICH_L |
| File Modified | 0000-00-00 |
| File Created | 2023-09-10 |