OMB Supporting Statement B THC Eval 11-29-16_clean

Supporting Statement B

Evaluation and Assessment of HRSA Teaching Health Centers

OMB Control No. 0915-0376

Extension

1. Respondent Universe and Sampling Methods

Statistical methods will not be used to create a sample group. For the Program Data Collection Tool, all programs receiving THCGME funding will be required to complete the form one time. There are currently 59 THC residency programs, and 10 programs have not yet completed the survey.

For the individual level surveys, responses will be critical for THC residency programs to meet the THCGME program annual reporting requirements. For example, the individual level surveys provide the THC residency programs a standardized way to collect background information and career outcome information on their residents and graduates. Background information, such as disadvantaged background, must be reported at the individual level for all HRSA funded THC residency programs. Career outcomes, such as caring for vulnerable populations living in underserved areas, are a legislated annual reporting requirement for the THCGME program.

The population of THC residents has currently reached its peak. In year 1 of the program (academic year 2011-2012) there were approximately 60 first-year THC residents who graduated in June, 2014 as primary care programs are generally three years. New programs were awarded in each subsequent year. Current programs reached full capacity in academic year 2015-2016 as approximately 200 residents per THC class matriculated. Therefore, the GW team expects 200 respondents per survey per year (i.e. in 2016 there were 200 respondents to the THC Matriculant Survey, 200 to the THC Graduation Survey, and an expected 200 to the THC Alumni Survey). Given the limited annual population size, the three survey instruments will be implemented with all THC residents to establish a census of the entire population of THC residents. All residents of THC programs will be surveyed at matriculation, graduation, and one-year post-graduation for the duration of the THC Evaluation contract with the George Washington University (GW). Analysis will be descriptive.

The current estimate of subjects for the individual level surveys is 600 per year. This may increase if additional programs are added. Response rates have been high as the THC programs are engaged with informing and encouraging their residents to respond. THC residency programs are incentivized to participate as the results will allow them to meet their own reporting requirements.

Annual Respondent Universe:

Form Name	Number of Respondents	Number of Responses per Respondent	Total Responses
Program Data Collection Tool	10 remaining	1	10 remaining
THC Alumni Survey	200	1	200
THC Matriculant Survey	200	1	200
THC Graduation Survey	200	1	200

2. Procedures for the Collection of Information

The standardized Program Data Collection Tool is implemented once a THC residency program has been in operation for at least one year and is completed only once. The Program Data Collection Tool collects administrative data on the residency program, including organization, training site, curriculum, amount of clinical service, health center, and aggregate resident information. There is no individually identifiable information on the Program Data Collection Tool. The instrument is implemented as a fillable pdf form delivered via e-mail to the THC primary contact person. Contact information will be provided by HRSA. Completed forms will be exported to a database. Analysis will be descriptive and provided back to HRSA as part of the requirement of the THC evaluation contract.

Three individual level surveys have been developed and are in use – a THC Matriculant Survey, a THC Graduation Survey, and a THC Alumni Survey. The THC Matriculant Survey gathers basic demographic information regarding the resident as well as information on their background, education, and career intentions. The THC Graduation Survey gathers program feedback, assesses preparedness to practice, information on career plans, and future contact information after graduation. The THC Alumni Survey gathers information on career outcomes (including location and type of practice, and further training), residency program, feedback, and updated contact information.

Surveys were developed based on survey instruments shared by existing THCs and other similar surveys, including the Washington, Wyoming, Alaska, Montana, Idaho (WWAMI) Family Medicine Residency Network survey and the Association of American Medical Colleges (AAMC) matriculant and graduate surveys. Specifically, the three surveys for the THC evaluation are based substantially on survey questions used by the Association of American Medical Colleges (AAMC), which is an association representing all 141 accredited US and 17 accredited Canadian medical schools. The AAMC implements a matriculating student and graduation questionnaire on an annual basis for all U.S. medical students in MD granting institutions and provides national level aggregated data. The AAMC does not currently make individual level data available to programs or researchers. The AAMC survey questions also are not specific to primary care, do not capture information on residency experience, do not provide an opportunity to provide feedback on a residency program, or collect information on career outcomes. For these reasons, we also modeled the THC residency survey after the WWAMI (Washington, Wyoming, Alaska, Montana, and Idaho) Family Medicine Residency Network, which has implemented a “Follow-Up Survey” for graduates (or alumni survey) every 2-3 years starting in 2000. This “Follow-Up Survey” allows the WWAMI residency programs to capture data on where their graduates are practicing, what fields they are practicing in and allows alumni residents to provide feedback to the WWAMI residency programs. The WWAMI Network includes 21 family medicine residency programs across five states, and the “Follow-Up Survey” is one of the most widely implemented resident level alumni surveys. However, the WWAMI Network survey is focused only on family medicine programs. Together, these surveys contain the most commonly used and validated questions to assess medical trainee backgrounds and career outcomes currently used in the U.S.

The questions in the THC Matriculant Survey, THC Graduation Survey and THC Alumni Survey are consistent with the AAMC and WWAMI Network surveys to allow some comparison. In some cases, questions were adapted to address differences in the timing of the survey instruments (at entry to residency rather than to medical school) or to reflect the different residency specialties that qualify for the THC program, including internal medicine, pediatrics, ob-gyn, psychiatry, geriatrics, and dental residents in addition to family medicine residents. In some cases, we have changed categorical responses to open ended responses (such as in questions on time spent between residency and medical school, and reasons for choosing the THC residency program) for ease of administration and to capture the most accurate and complete data from the THC residents. In other cases, we have expanded some response categories (such as questions on student debt, board certification and principle practice setting) in order to reflect the different time period (after medical school) when the survey will be implemented or to reflect the wider range of medical and dental specialties seen in the THC residency programs.

The THC Matriculant Survey, THC Alumni Survey, and THC Graduation Survey were also developed to correlate with the required HRSA performance measures for all Bureau of Health Workforce programs, including the THC residency programs. Specifically, we include questions in the THC Matriculant Survey, THC Graduation Survey, and THC Alumni Survey that are matched to HRSA Performance Measure requirements. These questions are responsive to the requirements of the program evaluation to document characteristics of the residents who participate in the HRSA funded residency program. For example, the THC Matriculant Survey includes a question on “disadvantaged background” that is consistent with requirements for HRSA performance measure reporting. The THC Matriculant Survey question on disadvantaged background includes the HRSA-approved definition of disadvantaged background to provide specific guidance for respondents. In cases where no validated questions were available, questions were developed from available research. A “rural background” question was developed to be consistent with HRSA performance measure requirements. However, in this case, HRSA provides a link to a website to determine if a county or address is considered rural. In this case, as supported by the 2007 study “Predicting rural practice using different definitions to classify medical school applicants as having a rural upbringing” by Owen et al., individuals are asked to self-identify “rural background” and provide high school information as an additional data point to assess rural background. See Table 1 for a crosswalk between survey questions, existing surveys, HRSA performance measures, evaluation questions, and relevant research literature.

The THC Matriculant Survey, the THC Alumni Survey, and the THC Graduation Survey were reviewed and piloted by four primary care physicians familiar with the AAMC surveys and the THC residency model, who provided feedback on length of time to complete the survey, the ease of on-line administration and clarity of the questions. Only minor modifications were made based on feedback.

Implementation of the THC Matriculant Survey, THC Graduation Survey, and THC Alumni Survey instruments has been done in collaboration with the THC residency programs. THC residency participation is voluntary; however, the survey instruments were designed to collect information that is part of program level required reporting for the HRSA funded programs. THCs are requested to provide resident e-mail addresses and inform residents and graduates that they will be receiving an e-mail from GW with a link to an online survey instrument. THCs have the option of informing residents in person or by e-mail. An e-mail template will be provided for the THCs (Appendix 1). All THC residency programs have been actively participating with all three individual surveys, and all have indicated positive feedback particularly with their data (residents from their program) that is shared with them by the GW evaluation team.

GW will directly contact residents and graduates via e-mail. The email for the Matriculant Survey is found in Appendix 2. The e-mail for the THC Graduation and Alumni surveys will contain the following: a link to an identifiable and secure online survey, using Survey Monkey; a link to an anonymous and secure online survey, using Survey Monkey; and an informed consent information sheet (Appendix 3). Consent will be inferred with completion of the survey. Two reminder e-mails (Appendix 4) will be sent to increase response rates, and THC residency programs will be asked to remind residents to complete surveys. The THC Matriculant Survey will be implemented in July of each year, at the start of the academic year. The THC Graduation and THC Alumni surveys will be implemented in June of each year, around commencement time. The Matriculant Survey will be sent to all new THC residents. Graduation surveys will be sent to all graduating residents. The THC Alumni Survey will be sent to all graduates at one-year post graduation.

The THC Matriculant Survey collects individually identifiable information because the HRSA performance measures require programs to report on an individual resident level (using a unique identifier created by the program). Data collected in the THC Matriculant Survey includes information on the resident’s background and career plans. Respondents will be informed that the information they provide will be provided back to their programs in an identifiable manner. All responses are voluntary and there is no penalty for non-response.

The THC Graduation and THC Alumni surveys collect both identified and anonymous data. The THC Alumni Survey and the THC Graduation Survey are separated into two parts: 1) the collection of identifiable data (career plan information and updated contact information); and 2) the collection of anonymous feedback on the THC residency program. Two separate links, with one link corresponding to the identified part and one link corresponding to the anonymous part, will be sent for each survey, along with an explanation of the two parts of the survey. Respondents will be informed that only the information they provide in the part covering career plans and updated contact information of the THC Alumni and THC Graduation surveys will be provided back to their programs in an identifiable manner. The respondents will be informed that the second part of the THC Alumni and Graduation surveys, which collects feedback on the residency program will be aggregated and de-identified before being provided to the THC residency program. All responses are voluntary and there will be no penalty for non-response.

A database of each THC’s residents and graduates’ responses will be provided back to the THC primary contact person. The THC Graduation Survey results will be held until after graduation to reduce any potential for identified data to affect individuals while still in residency.

Analysis will be descriptive. Practice addresses will be mapped to identify location in rural or other underserved areas. GW currently has one to two more years (through September 2017 or 2018) for the THC Evaluation contract with HRSA. GW will implement the survey instruments for the remainder of the contract. However, at the end of the contract period, the survey instruments will be provided back to the THC residency programs to continue collecting the information required for the HRSA funding.

3. Methods to Maximize Response Rates and Deal with Nonresponse

As the initial implementation of the residency surveys will create a baseline census for THC residents, response rates will be reported for all surveys and individual questions. There are no true comparison groups for this population; however, publicly available data from the nation-wide AAMC survey will serve as a comparison on certain resident characteristics. While national demographics exist for U.S. medical students, those choosing THC residency programs are expected to differ from national average medical student demographics. THC residents are expected to reflect characteristics of the programs they choose, such as geographic location and populations served.

Some analysis of non-responders will be possible. Response rates are expected to be highest for the THC Matriculant Survey due to the timing of the survey when individuals are starting residency training rather than approaching graduation or post-graduation. Therefore, any non-response on the THC Graduation Survey and THC Alumni Survey can be assessed for the characteristics of non-responders. Non-response can also be analyzed based on program characteristics, for example, if particular programs or types of programs have higher non-response than others.

Response rates for the WWAMI Network “follow-up for graduates survey” (or alumni survey) is generally 60-70%. Response rates for the AAMC questionnaires are generally around 80%. Response rates for the most recently completed round of THC surveys has been quite good. The Matriculant Survey achieved a rate of 92%; the Graduation Survey 89%, and the Alumni Survey 71%. We had expected a minimum 60% response rate and given the plan to partner with THC residency programs - generally small (around 8-10 residents total per year) with strong relationships with their residents and graduates – to encourage response, we had expected our response rates to be above 80% and surpassed this goal with the Matriculant and Graduation surveys. The THC Alumni Survey response rate also exceeds the WWAMI follow-up survey response rate and is far above national averages for external surveys that range from 10-15 percent.

GW will continue to engage the THC residency programs to inform and encourage the response of their residents and graduates as this strategy has been highly successful. THC programs are incentivized to participate because GW provides individual level survey responses back to the appropriate THC residency program of training in order for those THCs to meet their HRSA reporting requirements. THCs can either inform subjects in person, as they have daily contact with residents, or via e-mail. GW will also send two reminder e-mails to non-responders. As these are individual level resident and graduate surveys, participation is voluntary and there is no penalty for non-response.

4. Tests of Procedures or Methods to be Undertaken

The Program Data Collection Tool instrument was pilot tested with nine Teaching Health Centers in the first year of the THC Evaluation contract. The initial instrument was completed by nine THCs and feedback was requested after completion. The instrument was revised based on initial testing.

The THC Matriculant, THC Graduation, and THC Alumni surveys will be implemented over the remaining years of the THC Evaluation contract. At the end of the evaluation contract, it is expected that the THC residency programs will continue to use the survey instruments in order to meet the HRSA program reporting requirements.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

George Zangaro, Ph.D., tel. 301-443-9256, GZangaro@hrsa.gov, Director of National Center for Health Workforce Analysis, Health Resources and Services Administration

The person primarily responsible for data collection design, collection, and analysis is:

Marsha Regenstein, Ph.D., tel. 202-994-8662, marshar@gwu.edu, Department of Health Policy, Milken Institute School of Public Health, George Washington University

Appendix 1: Template E-mail from THC Programs to Residents/Graduates

Dear THC [resident/graduate],

As a [resident/graduate] of a Health Resources and Services Administration (HRSA) funded Teaching Health Center program, we are asking for your participation in an online survey to better understand the outcomes of the Teaching Health Center program. Your responses are critical to understand who the residents are who choose THC programs, to follow your career outcomes, and to provide feedback to improve our residency program.

In the next 1-2 weeks, you will receive an e-mail from our research partner, George Washington University, containing a link to an online survey. The survey is estimated to take no more than 10-15 minutes of your time. Participation in this survey is completely voluntary and there will be no penalties for non-response.

We are providing your contact email information to our research partner. This will be used only to initiate the survey contact and send up to two reminder e-mails. Your survey responses will be provided back to us, your residency program, in an identified manner. This is necessary in order for us to meet our required HRSA reporting for the THC funding. [Feedback on the residency programs shared in the Alumni and Graduation surveys will remain anonymous and only be provided to THCs in de-identified and aggregate form. Identifiable contact and career plan information in the Graduation Survey will not be released to the residency programs until after graduation.]¹

This project has been approved by the George Washington University Institutional Review Board and you will be provided with the details of the consent by them.

Thank you,

<Program Director Name>

Appendix 2: Subject Recruitment E-mail: THC Matriculant Survey

Dear THC [resident],

We request your participation in an online survey to better understand the outcomes of the Health Resources and Services Administration (HRSA) funded Teaching Health Center program. The survey collects basic background information, career plans and outcomes, and residency program feedback that will be critical to support, evaluate and improve the THC program.

Participation in this survey is voluntary. Your responses will be provided back to your residency program linked to your e-mail address – this is important for your residency program to be able to meet its reporting requirements for the HRSA funding. In most cases, survey questions collect basic information that bares little risk in sharing.

For publication purposes, survey results will be de-identified and aggregated across the THC programs. Your willingness to participate is implied if you proceed with completing this survey.

Please click on the following link for more information and to complete the survey:

<link>

Thank you for your participation.

Sincerely,

Marsha Regenstein, Ph.D.

Professor

Department of Health Policy

Milken Institute School of Public Health

The George Washington University

950 New Hampshire Avenue, NW, Suite 200

Washington, DC 20052

Office: (202) 994-8662

E-mail: marshar@gwu.edu

Appendix 3: Subject Recruitment E-mail: THC Alumni Survey and THC Graduation Survey

Dear THC [graduating resident/graduated resident],

We request your participation in an online survey to better understand the outcomes of the Health Resources and Services Administration (HRSA) funded Teaching Health Center program. The survey collects basic background information, career plans and outcomes, and residency program feedback that will be critical to support, evaluate and improve the THC program.

Participation in this survey is voluntary. Survey questions collect basic information on your background and career plans or outcomes. Graduation and alumni surveys also include some questions requesting your feedback on your residency program. This information is important to understand your experience in a unique training model and to help your residency program improve. Your background and career responses will be provided back to your residency program linked to your e-mail address – this is important for your residency program to be able to meet its reporting requirements for the HRSA funding. Your residency program feedback responses are anonymous and will be provided back to your residency program only in an aggregated and de-identified manner.

For publication purposes, all survey results will be de-identified and aggregated across the THC programs. Your willingness to participate is implied if you proceed with completing this survey. An informed consent information sheet is attached to this email.

Please click on the following link for more information and to complete [part 1] of the survey on your background and career plans or outcomes (your responses will be provided back to your residency program in an identifiable manner):

<link>

[Please click on the following link for more information and to complete part 2 of the survey on residency program feedback (your responses will be provided back to your residency program in an anonymous manner):

<link>]

Thank you for your participation.

Sincerely,

Marsha Regenstein, Ph.D.

Professor

Department of Health Policy

Milken Institute School of Public Health

The George Washington University

950 New Hampshire Avenue, NW, Suite 200

Washington, DC 20052

Office: (202) 994-8662

E-mail: marshar@gwu.edu

Informed Consent Information Sheet Attachment:

You are invited to take part in an evaluation study being conducted by Dr. Marsha Regenstein and Dr. Fitzhugh Mullan from the George Washington University Department of Health Policy. This project is being conducted under contract with the Health Resources and Services Administration (HRSA).

You are being asked if you want to take part in this study because of your position and experience with the {Insert THC Name here}. Your participation is completely voluntary, and you can withdraw at any time. Your academic standing/employment status/funding will not be affected in any way should you choose not to take part or to withdraw at any time.

The Teaching Health Center program was created through the Affordable Care Act of 2010 to support the expansion of new and existing primary care residency training programs in community-based ambulatory settings. The purpose of this study is to provide detailed quantitative and qualitative information about the structure, financing and operations of THCs and develop a framework for measuring their impact. Your program has been selected to be one of nine sites visited on this project. Ultimately, the outcomes of this study will inform policies on THC structure and funding, and improve physician workforce training to meet the needs of Americans.

This study will take about 1 hour of your time. During this hour, you will participate in an individual or panel interview. The researchers will ask you questions about your experience with {Insert name of THC here} and we will be audio recording all interviews to review later if needed. We encourage you to be open and honest in your responses. You are free to skip any questions or end the interview (or exit the panel) at any time. While we cannot guarantee the privacy of the panel group discussion, we request that all present respect the group by not discussing outside the group what is said. Contact information for the research team is provided at the top of the page and we will be available to discuss any items that, for privacy or other reasons, were not raised during the interviews.

For publication purposes, all data will be de-identified and reported in aggregate across the THCs. You will not be individually identified in any published articles or presentations, unless you specifically agree to be identified by name. As this work is being conducted under an evaluation contract with HRSA, responses will be reported to HRSA at the program level. We will not identify you by name in reports to HRSA. However, identified records may be provided to HRSA if requested. Your identified records for the study may also be reviewed by the George Washington University office responsible for overseeing research safety and compliance.

Taking part in this study will not benefit you directly, however the benefit to society will be a better understanding of how residency programs can better serve the nation by training doctors in the communities most in need of care. Please talk to the research team if you have questions, concerns, complaints, or think you have been harmed. You can contact the Principal Investigators listed on the top of this form at 202-994-8662 or 202-994-2994. For questions regarding your rights as a participant in human research call the GWU Office of Human Research at 202-994-2715.

Completion of the survey infers you have consented to participate in this study.

Appendix 4: Subject Reminder E-mail: THC Matriculant Survey, THC Alumni Survey and THC Graduation Survey

Dear THC [resident/graduate],

This is a reminder to please complete the Teaching Health Center survey.

The purpose of this survey is to better understand the outcomes of the Health Resources and Services Administration (HRSA) funded Teaching Health Center program. The survey collects basic background information, career plans and outcomes, and residency program feedback that will be critical to support, evaluate and improve the THC program.

Participation in this survey is voluntary. Survey questions collect basic information on your background and career plans or outcomes. Graduation and alumni surveys also include some questions requesting your feedback on your residency program. This information is important to understand your experience in a unique training model and to help your residency program improve. Your background and career responses will be provided back to your residency program linked to your e-mail address – this is important for your residency program to be able to meet its reporting requirements for the HRSA funding. Your residency program feedback responses are anonymous and will be provided back to your residency program only in an aggregated and de-identified manner.

Please click on the following link for more information and to complete [part 1]² of the survey on your background and career plans or outcomes (your responses will be provided back to your residency program in an identifiable manner):

<link>

[Please click on the following link for more information and to complete part 2 of the survey on residency program feedback (your responses will be provided back to you residency program in an anonymous manner):

<link>]¹

Thank you for you participation.

Sincerely,

Marsha Regenstein, Ph.D.

Professor

Department of Health Policy

Milken Institute School of Public Health

The George Washington University

950 New Hampshire Avenue, NW, Suite 200

Washington, DC 20052

Office: (202) 994-8662

E-mail: marshar@gwu.edu

1 Included in graduation and graduate survey e-mails.

2 Included only for graduation and graduate surveys

10