tribal HPOG_Supporting Statement Part B_Renewal_August 2014v4_clean

OMB Clearance Application

Evaluation of Tribal Health

Profession Opportunity Grants (HPOG)

Request for Revision on a Previously Approved Collection:
Control No: 0970–0395, Expiration 10/2014

Supporting Statement Part B

August 2014

2. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

• Site visits: Site visits will take place at each of the 13 sites (5 prime implementation sites; 8 secondary implementation sites) where the Tribal HPOG programs are being implemented; however, it is likely that data collection activities for secondary implementation sites outside of the prime grantee implementation site may be conducted via phone or teleconference, due to scheduling..

  • Interviews with Grantee and Partner Administrative Staff: Interviews will be conducted with the grant director and up to 6 additional staff members across primary and secondary implementation sites for each of the 5 grantee programs, for up to a total of 35 interviews annually. While respondents will represent a variety of positions across grantee programs, they will most likely include the project director and manager, staff from the workforce development and TANF partner organizations, public and private university-based partners, and not-for profit organizations. An advance letter (see Attachment A7) that describes the project, its goals, and the benefits of participation, will be sent to the grant director, and a recruitment letter will be used to recruit additional grantee and partner administrative staff, if necessary (see Attachment A8).

  • Interviews with Program Implementation Staff: Interviews will be conducted with staff responsible for implementing the Tribal HPOG programs in primary and secondary implementation sites. While respondents will represent a variety of positions due to the unique nature of each grantee program, they will most likely include instructors, recruitment and orientation staff, and providers of supportive services. Interviews will be conducted with up to 9 implementation staff at each primary and secondary implementation site, for a total of up to 117 interviews annually. Phone interviews may be conducted in lieu of in-person interviews with implementation staff at secondary implementation sites should scheduling conflicts arise. The universe of implementation staff will be identified by grantee administrative staff prior to initial site visits. While recruitment methods may vary by site, when possible we will send a recruitment letter signed by the grantee program (Attachment A8). We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling a time for the interview if appropriate.

  • Interviews with Employers: Interviews will be conducted with local or regional employers at public or private companies or organizations that are partnering with the Tribal HPOG program or have employed program participants. Interviews will be conducted with up to 4 employers at each primary and secondary implementation site, for up to a total of 52 interviews annually. Phone interviews may be conducted in lieu of in-person interviews with employers at secondary implementation sites should scheduling conflicts arise. This total may vary based on the number of employers located and identified by grantees at each distinct site. While recruitment methods may vary by site, when possible we will send a recruitment letter signed by the grantee program (Attachment A8). We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling a time for the interview if appropriate.

  • Focus Groups or Interviews with Program Participants: Focus groups or interviews will be conducted with current student program participants across all primary and secondary implementation sites. Based on our experience with the focus groups and interviews already conducted, up to 9 individuals will participate in focus groups or interviews) in each primary and secondary implementation site, for a total of up to 117 program participant respondents. Grantees will provide a list of all current participants by implementation site prior to each round of site visits. The tribal evaluation team will work closely with grantees to identify the appropriate recruiting mechanism for a targeted group of respondents. While recruitment methods may vary by site, when possible we will first send potential focus groups and interview participants a recruitment letter signed by the grantee (see Attachment A9 for focus group recruitment letter; see Attachment A10 for interview recruitment letter). Depending on the number of current participants in each implementation site, we may send recruitment letters to the entire pool of current participants, or randomize the lists of participants and recruit in waves. The use of random sampling has been employed as a sampling strategy in similar qualitative research efforts¹. We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and communicating additional information, such as meeting date, time, and place, as appropriate. In other cases, we may work with the grantee site to distribute a flyer to program participants (Attachment A11).

• Telephone Interviews with Program Completers: Telephone interviews will be conducted with student participants across all primary and secondary implementation sites who completed the Tribal HPOG programs. Up to 15 program completers in each grantee program will participate in interviews, for a total of up to 75 interviews, respectively². At least 4 interviews will be conducted in each implementation site. Grantees will provide a list of all program completers by implementation site after the site visits. The tribal evaluation team will work closely with grantees to identify the appropriate recruiting mechanism for a targeted group of respondents. While recruitment methods may vary by site, when possible we will first send potential interview participants a recruitment letter signed by the grantee (Attachment A10). Depending on the number of program completers in each implementation site, we may send recruitment letters to the entire pool of completers, or randomize the lists of completers and recruit in waves. As noted above, the use of random sampling has been employed as a sampling strategy in similar qualitative research efforts³.We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling an interview, as appropriate.

• Telephone Interviews with Program Non-Completers: Telephone interviews will be conducted with participants that did not complete the Tribal HPOG program. This number may vary across sites. The grantees will provide a list of student non-completers across implementation sites every other month starting after the initial site visits. Up to 4 interviews will be conducted with non-completers in each grantee program for a total of up to 20 interviews. While recruitment methods may vary by site, when possible we will first send potential interview participants a recruitment letter signed by the grantee (Attachment A10). Depending on the number of program completers in each implementation site, we may send recruitment letters to the entire pool of completers, or randomize the lists of completers and recruit in waves. As noted above, the use of random sampling has been employed as a sampling strategy in similar qualitative research efforts⁴. We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling an interview, as appropriate. Alternatively, upon notice to the grantee of a participant’s intent to withdraw or de-matriculate from the program (which is a standard procedure in institutions of higher or vocational education), the program coordinator (or designate) will provide the recruitment letter to all program non-completers.

2. Procedures for Collection of Information

No statistical methods will be used for stratification and sample selection.

The table below provides an estimated timeline of data collection activities. Funding for Tribal HPOG grantees began on September 30, 2010 and ends on September 29, 2015. This table details the timeline of data collection activities completed in the second, third and fourth years, as well as the timeline for data collection activities proposed for the final year of the evaluation..

Timeline of Data Collection Activities

Years 2-4 Activities	Completion Date
Conducted Initial Site Visits In-person interviews with Grantee and Partner Administrative Staff In-person interviews with Program Implementation Staff In-person interviews with Employers Focus groups with Program Participants	02/12-11/12
Conducted Follow-up 1 Site Visits In-person interviews with Grantee and Partner Administrative Staff In-person interviews with Program Implementation Staff In-person interviews with Employers Focus groups with Program Participants	02/13-10/13
Conduct Interviews with Program Completers	03/12-12/12
Conduct Interviews with Program Non-Completers	03/13-12/13
Year 5 Activities	Expected Date
Conduct Follow-up 2 Site Visits In-person interviews with Grantee and Partner Administrative Staff In-person interviews with Program Implementation Staff In-person interviews with Employers Focus groups with Program Participants	1-6 months following OMB approval
Conduct Interviews with Program Completers	5-7 months following OMB approval
Conduct Interviews with Program Non-Completers	5 to 7 months following OMB approval

• Site Visits

Information for the Grantee and Partner Administrative Staff, Program Implementation Staff, Employers interviews and Program Participant focus groups will be collected by members of the tribal evaluation team during site visits (or by telephone with respondents located in secondary implementation should scheduling conflicts arise). The tribal evaluation team will prepare an advance letter that describes the project, its goals, and the benefits of participation, and send it to the grantee directors (see Attachment A7). Site visits will be conducted at 1-6 months following OMB approval.

Interviews with Grantee and Partner Administrative Staff will be conducted to gain insight on high-level program strategies, program development and lessons learned. Interviews with Program Implementation Staff will shed light on program processes including recruitment; orientation and program implementation. Interviews with Employers will provide information on employers’ general impressions of program graduates, their degree of awareness of the program as a whole, and their views on the extent to which programs are making an impact on the creation of a workforce that is equipped to meet the demands of the current health care needs of tribal communities. All interviews are constructed in a modular fashion so that different sets of questions can be combined according to the role of the respondent and topics of interest.

Focus groups with Program Participants will be conducted to gain program participants’ perceptions around program design and curriculum; supportive services; recruitment and orientation; incorporation of Family Education Model (FEM); quality of instruction; educational attainment; and satisfaction with the Tribal HPOG program..

Tribal HPOG grantees will provide the tribal evaluation team with a comprehensive list of grantee and partner administrative staff, staff responsible for implementing the program, local employers, and current participants and their contact information. In consultation with each Tribal HPOG grantee, the tribal evaluation team will recruit using recruitment letters or flyers and if necessary, follow-up phone calls (see Attachments A8, A9 and A10 for recruitment letters and Attachment A11 for flyer). Should the Tribal HPOG grantee prefer to have potential participants contacted by program staff (e.g., the Program Coordinator) we will provide the recruitment materials as described to facilitate recruitment. All interviews will be conducted by appointment.

Each site visit will be three to five days in length. Interviews will be conducted by at least two team members, one senior staff member to conduct the interview and a junior staff member to take comprehensive notes and identify areas to be covered as the interview progresses. Interviews will be conducted using detailed protocols and note-taking guides that are designed to offer maximum flexibility in gathering input from the respondents. Assuming that meeting logistics are well coordinated with attention to what will be the most convenient time for focus group participants is crucial to support adequate participation and, importantly minimize burden on participants. In all cases, we will work with grantees to identify the best time and venue for conducting focus groups. Following each site visit, the site visit team will meet with the rest of the project team to conduct a formal debriefing of the findings.

• Program Completers

Information concerning Program Completers will be collected through telephone interviews with participants that have completed the Tribal HPOG program. These interviews will be used to obtain information on key program outcomes related to educational attainment and employment. These interviews will occur 5-7 months after OMB approval. We anticipate that some participants in the focus groups will also participate in the interviews; others may have had no previous contact with the evaluation team.

Grantees will provide a list of all program completers by implementation site after the site visits. While recruitment methods may vary by site, when possible we will first send potential interview participants a recruitment letter signed by the grantee (Attachment A10). We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling an interview, as appropriate. Interviews will be conducted by at least two team members, one senior staff member to conduct he interview and a junior staff member to take comprehensive notes and identify areas to be covered as the interview progresses. Following the conclusion of each interview, the tribal evaluation team will produce a clear, electronic copy of interview notes that summarize key findings.

• Program Non-Completers

Information concerning Program Non-Completers will be obtained through telephone interviews with participants that did not complete the Tribal HPOG program. The purpose of these interviews will be to learn about reasons for leaving the program and challenges experienced, elements of the program that were effective or non-effective and identification of any outcomes resulting from program participation.

While recruitment methods may vary by site, whenever possible upon notice to the grantee of a participant’s intent to withdraw or de-matriculate from the program (which is a standard procedure in institutions of higher or vocational education), the program coordinator (or designate) will provide a recruitment letter supplied by the tribal evaluation team which states the purpose of the interview, addresses respondent privacy, provides contact information about the evaluation team and human subject protections, and details the procedures for receiving an incentive for participation (see Attachment A10). In addition, grantees will provide the tribal evaluation team with a list of non-completers, including contact information, every other month starting after the site visits. Non-completers will likely have very limited contact information, limited telephone access or may have moved from the area. A member of the tribal evaluation team will then attempt to contact each non-completer by phone to ask if he or she is willing to participate in the exit interview. As with the participant interviews, the evaluation lead will seek verbal informed consent.

All members of the evaluation team will be trained to conduct the site visits, administer the interviews, and facilitate focus groups in a culturally competent manner. Interviewers will also be trained on their responsibilities to ensure privacy. The content of the interview protocols and focus group guides were developed by tribal evaluation team partners in consultation with key stakeholders, including the Tribal HPOG grantees, the internal Advisory Committee, ACF officials, and shared with the HPOG evaluators.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Gaining cooperation and buy-in from grantees are key objectives of this data collection effort. For this data collection effort, we believe that the specialized nature of the respondent groups – grantees and partners, and in some cases implementation staff and employers - that received funding from ACF and who are interested in supporting HHS efforts to design and implement health professions training programs will increase their propensity to respond, resulting in a response rate of at least 80 percent. It is important to note that the tribal HPOG grantees are aware of this assessment and have agreed to participate in the assessment as part of their cooperative agreement with ACF. To date, we have been very successful in gaining cooperation and buy-in from grantees, and have had very few refusals for participation during the onsite interviews and focus groups (response rate is estimated as close to 90 percent).

In the case of the participant focus groups and interviews, the tribal evaluation team will work with the tribal HPOG grantees to recruit these individuals. Given that the participants are likely involved with the Tribal HPOG program or are receiving its services, it will be effective to collaborate with the Tribal HPOG grantees to secure the participation of these individuals.

In addition, the tribal evaluation team will use a number of proven methods to maximize participation and cooperation in the study:

• Advance and Recruitment Letters: Advance and/or recruitment letters will be sent to all potential respondents. A description of the evaluation will be included in this mailing. Follow-up introductory calls will be made with prospective respondents to answer questions and address concerns.

• Pre-interview preparation: Grantee and Partner Administrative Staff, Program Implementation Staff, and Employers will be provided topics that will be covered in the interview prior to the interviews to gain cooperation and minimize burden during the actual interview.

• Administration: Interviews conducted during the site visits will be scheduled at the respondents’ convenience. Focus groups will be conducted during the lunch or evening hour and light refreshments and transportation vouchers will be provided. Telephone interviews with program participants (completers and non-completers) will be scheduled at the respondents’ convenience.

• Assurances of privacy: Respondents will be assured that reported data are aggregated and not attributable to individuals or organizational entities.

• Use of Tokens of Appreciation: A token of appreciation valued at $50 will be provided to the Program Participants (focus groups). Program Participants (at primary and secondary implementation sites) and Program Completers that participate in the telephone interviews in lieu of the focus groups will receive a non-cash token of appreciation valued at $25. Non-completers that participate in telephone interviews will receive a non-cash token of appreciation valued at $10. Each grantee will be consulted to determine the most appropriate method of payment (e.g., gift certificate to a local grocery store).

While we expect a few hard refusals to the interviews and focus groups, we have found that these techniques are highly effective ways to increase response rates, particularly in cases where the overall sample size is small enough to provide a “personalized” or high-touch level of follow-up. As mentioned above, we have had minimum refusals for participation in the onsite interviews and focus groups. We have been less successful in obtaining participation for the completer and non-completer interviews; however, this is not due to refusal. Contact information (email and phone) is outdated and therefore, we cannot reach all of the potential completers and non-completers. We will continue to work with the grantees on this and provide the recruitment letters to increase participation in the completer and non-completer interviews.

4. Tests of Procedures or Methods to be Undertaken

The final instruments have been reviewed by (1) Office of Information and Regulatory Affairs, Office of Management and Budget, (2) Administration for Children and Families staff, (3) A group of consultants comprised of workforce and tribal researchers, (4) staff at the National Indian Health Board staff, a tribal membership organization and tribal employer, and (5) staff from Red Star Innovations, a tribally owned small business. Their comments were incorporated into the final versions.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and /or Analyzing Data

The information for this study is being collected by NORC at the University of Chicago, a non-profit social science research organization, and its subcontractors, the National Indian Health Board and Red Star Innovations, on behalf of the US Department of Health and Human Services (HHS). With HHS oversight NORC and its subcontractors are responsible for the collection and analysis of qualitative data and producing site visit reports, practice briefs, an interim report, and a final report. Key input into the design was received from the following individuals:

NORC at the University of Chicago team

Current Team Members

Michael Meit, Project Director, NORC at the University of Chicago

Alana Knudson, Senior Advisor, NORC at the University of Chicago

Carol Hafford, Co-Task leader, NORC at the University of Chicago

Aleena Hernandez, Co-Task Leader, Red Star Innovations

Katherine Meyer, Project Manager, NORC at the University of Chicago

Shannon TenBroeck, Principal Research Analyst, NORC at the University of Chicago

Jennie Alfaro, Senior Research Analyst, NORC at the University of Chicago

Tess Gilbert, Senior Research Analyst, NORC at the University of Chicago

Noelle Miesfeld, Research Analyst, NORC at the University of Chicago

Robert Foley, National Indian Health Board

Previous Team Members

Jessica Bushar, Project Manager, NORC at the University of Chicago

Hilary Scherer, Research Analyst, NORC at the University of Chicago

Heather Langerman, Research Assistant, NORC at the University of Chicago

Evangeline Dotomain, National Indian Health Board

Consultants

Matthew L. Boulton, MD, MPH

Mark Doescher, MD, MPH

Kristie Gebbie, DrPH, RN

Jacque Gray, PhD

Felicia Schanche Hodge, DrPH

Hugh Tilson, MD, DrPH

Department of Health and Human Services, Administration for Children and Families

Hilary Forster, Project Officer, Office of Planning Research, and Evaluation

Molly Irwin, HPO Evaluation Team Lead, Office of Planning Research, and Evaluation

Tribal Health Professions Opportunities Grants grantees

Current Tribal HPOG Project Directors

Alfred DeRoche, Blackfeet Community College

Mark Hiratsuka, Cook Inlet Tribal Council

Teresa Halcsik, College of the Menominee Nation

Irene BearRunner, Turtle Mountain College

Phillip Longie, Cankdeska Community College

Previous Tribal HPOG Project Directors

Terry Tatsey, Blackfeet Community College

Carol Wren, Cook Inlet Tribal Council

Kimberly O’Leary, College of the Menominee Nation

Larretta Hall, Turtle Mountain College

Cynthia Lindquist-Mala, Cankdeska Community College

The HHS Project Officers, Michael Dubinsky and Hilary Forster, have overseen the design and can be contacted at:

Hilary Forster

Office of Planning, Research, and Evaluation

Administration for Children and Families

US Department of Health and Human Services

370 L’Enfant Promenade, SW

Washington DC 20477

(202) 619-1790

Data collection and analysis has been conducted by Michael Meit, Alana Knudson, Carol Hafford, Jessica Bushar, Hilary Scherer, and Heather Langerman, Katherine Meyer, Tess Gilbert and Jennie Alfaro, all of NORC at the University of Chicago, Aleena Hernandez of Red Star Innovations, and Evangeline Dotomain and Robert Foley of National Indian Health Board.

For the purposes of this upcoming data collection, data collection and analysis will be conducted by Michael Meit, Alana Knudson, Carol Hafford, Katherine Meyer, Shannon TenBroeck, Tess Gilbert and Noelle Miesfeld, all of NORC at the University of Chicago, Aleena Hernandez of Red Star Innovations, and Robert Foley of National Indian Health Board.

1 Huston, A.C., Miller, C, Richburg-Hayes, L., Duncan, G.J., Eldred, C.A., Weisner, T.S., et al. (June 2003). New Hope for Families and Children: Five-Year Results of a Program to Reduce Poverty and Reform Welfare. NY: MDRC.

2 The reduction in numbers is not due to nonresponse; rather, it is due to contact information (email and phone) that is out of date and we cannot contact the students, resulting in a smaller group of students to contact for interviews.

3 Huston, A.C., Miller, C, Richburg-Hayes, L., Duncan, G.J., Eldred, C.A., Weisner, T.S., et al. (June 2003). New Hope for Families and Children: Five-Year Results of a Program to Reduce Poverty and Reform Welfare. NY: MDRC.

4 Huston, A.C., Miller, C, Richburg-Hayes, L., Duncan, G.J., Eldred, C.A., Weisner, T.S., et al. (June 2003). New Hope for Families and Children: Five-Year Results of a Program to Reduce Poverty and Reform Welfare. NY: MDRC.

1