NOW IS THE TIME (NITT) – MINORITY FELLOWSHIP PROGRAM (MFP) EVALUATION
SUPPORTING STATEMENT
The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Mental Health Services (CMHS) is requesting approval from the Office of Management and Budget (OMB) for new data collection activities for SAMHSA’s Evaluation of the MFP – Youth and MFP – Addiction Counselors programs, part of the Now Is The Time (NITT) initiative. SAMHSA’s NITT initiative aims to better protect U.S. children from violence by making schools safer and increasing access to mental health services. Programs included are NITT Project AWARE, Healthy Transitions and MFP (Youth and Addiction Counselors). While SAMHSA’s NITT initiative includes separate programs, these programs are united by their focus on capacity building, system change and workforce development. The NITT national evaluation is scheduled through March 1, 2020 and will conduct a national cross program evaluation of all three grant programs.
The data collection activity described in this package pertains exclusively to the telephone interviews that will be conducted for the NITT-MFP program components of the national evaluation. SAMHSA is requesting approval for the following data collection: the NITT-MFP Fellow Interview, which will collect detailed information from a subset of fellowship participants on their motivations to apply for the program, the skills and information they learned as fellows, and the influence of the program on their career plans. (Attachment 1)
SAMHSA’s NITT-MFP – Youth program is authorized under Section 520A of the Public Health Service Act, as amended, and addresses the Healthy People 2020 Mental Health and Mental Disorders Topic Area (HP 2020-MHMD) and Substance Abuse Topic Area (HP 2020-SA). SAMHSA’s NITT-MFP – Addiction Counselors program is authorized under Section 509 of the Public Health Service Act, as amended, and addresses the Healthy People 2020 Substance Abuse Topic Area (HP 2020-SA). Both NITT-MFP programs also support two of SAMHSA’s Strategic Initiatives: (1) Prevention of Substance Abuse and Mental illness, and (2) Workforce Development. Finally, this effort supports the fourth component of the President’s NITT Plan, to increase access to mental health/behavioral health services.
The President’s NITT initiative responds in part to the fact that many children, youth, and young adults who need behavioral health services do not receive them, and this is particularly so for minority populations. Racial and ethnic minorities have less access to behavioral health services than whites and, when they do receive behavioral health treatment, it is more likely to be of poor quality. A key component in reducing disparity in access to behavioral health services is attention to behavioral health workforce development. The behavioral health workforce is inadequate to address the expanding needs of U.S. children, youth, and young adults. There are not enough behavioral health providers, particularly in rural communities, and inadequate resources to support behavioral health provider training and ongoing supervision (IOM, 2006). Several recent changes will increase the need for an adequate supply of skilled behavioral health workers: 1) recent policies will increase the number of people with coverage for behavioral health care; 2) increased use of screening in primary care settings will identify more behavioral health needs; 3) many returning veterans will need services; and 4) state initiatives to reduce prison populations will introduce more individuals with behavioral health needs back into communities (SAMHSA, 2013).
The current behavioral health workforce is aging and large numbers will retire in the next few decades (SAMHSA, 2013). However, it is difficult to recruit and retain highly qualified individuals into these occupations because the compensation is low relative to the high job stress and heavy workloads (Bureau of Labor Statistics [BLS], 2012), and there is prejudice associated with mental health and substance abuse occupations (IOM, 2006). The high turnover rates in behavioral health jobs are also attributed to lack of clear professional pathways or career ladders, and the absence of effective supervision to mentor, lead, and develop staff (Hoge, et al., 2007). BLS projects that between 2012 and 2022, an additional 28,000 Substance Abuse and Behavioral Disorder Counselors, and an additional 48,200 Mental Health Counselors and Marriage and Family Therapists will be needed (BLS, 2012). Without proactive measures, the current deficits across behavioral health professional disciplines will worsen.
The urgency and the challenges for training and retaining a well-prepared behavioral health workforce were thoroughly articulated in An Action Plan for Behavioral Health Workforce Development (Hoge, et al., 2007). This action plan highlighted the increasing prevalence of co-occurring mental and addictive disorders in the U.S., coupled with the absence of training to address co-occurring disorders provided to the behavioral health workforce, as particularly problematic. The action plan also cited the lack of cultural competency skills and training in this workforce, a gap made wider by the disproportionally small number of racially and ethnically diverse behavioral health professionals relative to U.S. demographics. As demographics change and the proportion of minority groups increase from 37% to a projected 57% of the U.S. population by 2060 (U.S. Census Bureau, 2012) the need for a culturally competent behavioral workforce will become increasingly important (Hoge, et al., 2013). In fact, the action plan concluded that mental health providers are “too often insensitive to the needs of individuals, as these are affected by ethnicity, culture, and language” (Hoge, et al., 2007). As the lead federal agency responsible for improving the services provided by the mental health and addictions workforce, SAMHSA has put significant effort into understanding the behavioral workforce issues and developing plans to address them, including implementation of NITT-MFP.
The traditional MFP was established in 1974 to provide specialized training to mental health psychiatrists, nurses, social workers, and psychologists to improve services provided in underserved minority communities. The traditional MFP has resulted in a significant rise in the number, capacity and multidisciplinary leadership presence of minorities throughout the behavioral health community. Over the past four decades, the program has expanded to support more professional disciplines including marriage and family therapists and professional counselors. In 2014, funds were appropriated to expand the traditional MFP to include NITT-MFP to provide stipends and tuition support to students pursuing Master’s level training in behavioral health fields like psychology, social work, professional counseling, marriage and family therapy, nursing, and addiction/substance abuse counseling, thus directly supporting the NITT goal of increasing behavioral health services for youth and contributing to making schools safer.
NITT–MFP – Youth funded five grantees to each support up to 48 Master’s level fellows per year committed to addressing the behavioral health needs of at risk children, adolescents, and populations transitioning to adulthood (ages 16–25). By recruiting and providing specialized training to psychologists, nurses, family therapists, social workers, and counselors who will focus their services on underserved minority youth, NITT MFP – Youth is intended to increase the skilled workforce providing mental health services where they are needed the most. NITT–MFP – Addiction Counselors funded two grantees to each support a maximum of 30 Master’s fellows per year in their final year of addiction counseling university programs, with a focus on providing culturally sensitive addiction counseling to underserved youth in the 16–25 age group, which has higher substance use rates than any other age group (SAMHSA, 2013). This support will help to fill another important need in the behavioral health workforce. The NITT-MFP programs provide one year fellowships.
Table 1 provides an overview of the number of grantees per NITT-MFP program, the number of fellows each grantee will support per cohort year, the total number of fellows supported each cohort year, the number of cohorts, and the total number of fellows that will be supported over the four-year life of the grant programs.
Table 1. Now is the Time (NITT) – Minority Fellowship Program (MFP) Grantee and Fellow Overview
NITT-MFP Program |
Number of Grantees |
Fellows per Year |
Total Fellows per Year |
Number of Cohorts |
Fellows Total |
Youth |
5 |
48 |
240 |
4 |
960 |
Addiction Counselors |
2 |
30 |
60 |
4 |
240 |
TOTAL |
7 |
78 |
300 |
|
1,200 |
The evaluation of the NITT-MFP programs will assess the overall success of the SAMHSA NITT initiative in enhancing the behavioral health workforce in terms of the number of new graduates from the NITT-MFP programs, their competencies and characteristics, and their capacity to meet behavioral health workforce needs. The NITT-MFP evaluation will collect data on changes in the number of culturally competent behavioral health professionals available to provide clinical services to underserved, at risk children, adolescents and youth ages 16–25 transitioning to adulthood. The evaluation will also explore whether the program results in increased knowledge, skills, and aptitude among NITT-MFP fellows to provide culturally competent behavioral health services to these populations; the costs associated with increasing the behavioral health workforce; and how these new behavioral health professionals are sustained in the workforce.
The NITT-MFP evaluation is designed to assess the level of success of the grantees in meeting the programs’ goals and identifies the factors that contribute to differences among grantees in levels of success. These research questions will be addressed by the NITT-MFP process and outcome evaluation. The process evaluation will include both qualitative and quantitative components supported by multiple types of data collection—reviews of grantee application and program documentation, interviews with fellows, and the analysis of survey data from fellows and program alumni (approved under OMB No. 0930-0304). The NITT-MFP outcome evaluation includes two levels of primary outcomes: grantees and fellows. The Web-based surveys of current fellows and fellowship program alumni have already received OMB clearance (OMB No. 0930-0304); this NITT-MFP OMB package focuses on the telephone interviews with selected fellows. The outcome evaluation will focus on how differences in professional disciplines and grantees’ program strategies contribute to observed differences in fellow outcomes.
The NITT-MFP evaluation questions to assess program processes and outcomes fall into two broad areas: 1) mental health service system capacity and infrastructure, and 2) early and accurate identification of mental health needs. Table 2 describes the NITT-MFP evaluation questions; questions 1-5 address priority area 1, and question 6, priority area 2.
Table 2. Evaluation Questions for the Now is the Time (NITT) - Minority Fellowship Program (MFP) Programs
EQ1. |
To what extent does NITT-MFP increase the number of culturally competent behavioral health professionals available to provide clinical services to underserved, at-risk children, youth, and young adults in public and private settings? |
EQ2. |
To what extent does NITT-MFP prepare fellows for the supervisory behavioral health workforce? |
EQ3. |
To what extent does NITT-MFP build the supervisory behavioral workforce for behavioral professionals that provide clinical services to underserved, at-risk, children, youth, and young adults? |
EQ4. |
What costs are associated with increasing the behavioral health workforce and how do the costs vary across professional disciplines? |
EQ5. |
What are the intended/actual/projected ways in which new behavioral health professionals will be sustained in the workforce? |
EQ6. |
To what extent are the knowledge, skills, and aptitude of fellows to provide culturally competent behavioral health services to underserved, at-risk children, youth, and young adults increased as a result of the NITT-MFP? |
The data collected through the NITT-MFP Fellow Interviews will provide detailed, descriptive information about the interviewees’ experiences in the NITT-MFP program and their assessment of the program’s impact on their professional interests and career choices. The information collected through the interviews, in conjunction with the survey data provided under OMB No. 0930-0304, will provide the data necessary to conduct a process and outcome evaluation of the NITT-MFP programs, as described above.
The NITT-MFP Fellow Interview (Attachment 1) will collect detailed qualitative information on fellows’ experiences that is not possible to collect in a survey. The interview is timed to collect fellows’ impressions of their fellowship experiences before too much time has passed, as well as their initial labor market outcomes. The information collected will be used to assess the NITT-MFP program factors associated with employment and other post-fellowship outcomes. A subset of fellowship recipients will be recruited to complete the telephone interview approximately 4 to 5 months post program completion. The interviewees will be asked to describe 1) their program, how they learned about it, and what led them to apply; 2) the role of the fellowship stipend in their ability to participate in the program; 3) the effects of the program on their interest in working with at risk children, adolescents, and transition age youth from racially and ethnically diverse backgrounds (and for MFP-AC fellows, in the area of addiction counseling); and 4) whether the program improved their understanding of and ability to provide culturally competent services. In addition, the interview will ask about fellows’ program and post-program outcomes, including 1) whether they completed their fellowship and the effects of the stipend on their education and career, 2) their current employment setting, and if in behavior health services, the characteristics of their client population, 3) the role that their fellowship played in their job interests and job search, and 3) their satisfaction with the fellowship program and their assessment of its impact on their career and professional activities. This information is necessary to assess whether the program is having the intended effects of increasing participants’ interest and willingness to become behavioral health providers and addiction counselors providing services to underserved children, adolescents, and transition-age youth (ages 16-25), and particularly among racially/ethnically diverse populations.
An overview of the NITT-MFP evaluation questions and the items from the Fellow Interview that will be used to measure them are listed in Table 3. Evaluation question 4 is excluded because the data needed to answer that question will be collected through information supplied by the seven grantees.
Table 3. Now Is the Time (NITT) - Minority Fellowship Program (MFP) National Evaluation – Evaluation Questions and the Fellow Interview
EQ1. To what extent does NITT-MFP increase the number of culturally competent behavioral health professionals available to provide clinical services to underserved, at-risk children, youth, and young adults in public and private settings? |
|
Instrument |
Items |
NITT-MFP Fellow Interview |
12-22 |
EQ2. To what extent does NITT-MFP prepare fellows for the supervisory behavioral health workforce? |
|
Instrument |
Items |
NITT-MFP Fellow Interview |
10 |
EQ3. To what extent does NITT-MFP build the supervisory behavioral workforce for behavioral professionals that provide clinical services to underserved, at-risk, children, youth, and young adults? |
|
Instrument |
Items |
NITT-MFP Fellow Interview |
10 |
EQ5. What are the intended/actual/projected ways in which new behavioral health professionals will be sustained in the workforce? |
|
Instrument |
Items |
NITT-MFP Fellow Interview |
9-11; 22 |
EQ6. To what extent are the knowledge, skills, and aptitude of fellows to provide culturally competent behavioral health services to underserved, at-risk children, youth, and young adults increased as a result of the NITT-MFP? |
|
Instrument |
Items |
NITT-MFP Fellow Interview |
12-22 |
The NITT-MFP Fellow Interview will be conducted by telephone by members of the evaluation team. With respondent consent, interviews will be recorded as a back-up to the note taker. After the interview, the interviewer and note taker will review the completed interview notes for accuracy; any areas of discrepancy will be validated with the recording. Once the interview responses are considered final, the recording will be deleted. Until they are deleted, the recordings will be kept by the note taker on a secure, password protected server.
To avoid the duplication of data collection efforts, the NITT-MFP evaluation will not administer any surveys to MFP fellows. Instead, the information collected through the fellow interviews will be supplemented by data from two surveys (of current fellows and alumni) administered by another contractor under OMB No. 0930-0304. The NITT-MFP evaluation Fellow Interview will collect information unique to NITT-MFP programs that is otherwise not available, with the exception of the first eight questions in the interview. These questions are repeated in the interview to collect richer and more detailed information using open response options and provide the opportunity to ask for additional information. In addition, the response rate for the survey is not 100 percent, so some of the questions are repeated to ensure that information on the fellows’ programs that is essential for interpreting the interview results (e.g., confirming which program the fellowship was funded under) is available for all of the interviewees.
The surveys are designed to track program outputs, such as the number of fellows and their employment status over time. The fellow interviews are designed to expand upon the questions asked through the survey and provide detailed and in-depth information that will aid in the interpretation of the survey results. SAMHSA, for example, is specifically interested in understanding the training and mentoring strategies utilized by grantees to develop the cultural competency of fellows to address the behavioral health needs of underserved, racially/ethnically diverse, transition age youth (16-25). The interviews will provide information on this and other aspects of the fellowship program that SAMHSA has determined are not addressed by the other surveys.
Participation in this evaluation will not impose a significant impact on small entities. Data collection respondents are fellowship recipients.
The interviews will be conducted only one time with each respondent.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on December 28, 2015 (80 FR 80783). No comments were received.
SAMHSA has made extensive use of experts in the areas of young adult behavioral health, workforce development, and health disparities research and current grantee staff to provide guidance on the design and analysis of the NITT-MFP evaluation. The following NITT-MFP virtual meetings were held to review the various aspects of the NITT-MFP evaluation, including the preliminary evaluation plan and data collection procedures which were distributed prior to each meeting to give participants the opportunity to review in preparation for the meeting.
Introductory Webinar (July 2015)
Grantee Virtual Meeting (August 2015)
Expert Panel Meeting (August 2015)
The grantees have had several opportunities to ask questions about and provide feedback on the preliminary evaluation plan and data collection procedures. Feedback was provided on all aspects of the evaluation and participants’ comments and suggestions were incorporated into the development of the data collection procedures and instruments. Feedback included:
Create opportunities for NITT-MFP grantees to review the interim findings from the evaluation and review data.
Coordinate contacts with NITT-MFP fellows with the grantees and other entities collecting information from the fellows.
Offer grantees the opportunity to review the interview instrument.
The list of expert panel members is provided in Table 4.
Table 4. Now is the Time – Minority Fellowship Program Evaluation Expert Panel
Expert |
Affiliation |
Contact Information |
MaJosé Carrasco, M.P.A |
Director |
Email: majose@nami.org |
Lynda Gargan, Ph.D., M.Ed. |
Interim Executive Director |
Phone: 240-403-1901 Email: lgargan@ffcmh.org |
Susan Keys, Ph.D. |
Associate Professor and Senior Researcher |
Phone: 541-322-2046 Email: susan.keys@osucascades.edu |
Benjamin Le Cook, Ph.D., M.P.H. |
Assistant Professor of Psychiatry |
Phone: 617-503-8449 Email: bcook@cha.harvard.edu; bcook@charesearch.org |
Pamela Orpinas, Ph.D. |
Professor |
Phone: 706-542-4372 Email: porpinas@uga.edu |
Mark Salzer, Ph.D. |
Professor and Chair |
Phone: 215-204-7879 Email: mark.salzer@temple.edu |
Fellows who agree to be interviewed will receive a gift card respondent incentive worth a $30 value. The fellowship alumni may be difficult to reach because they are new and typically young professionals who are establishing their professional lives, which may entail frequent changes of workplace and residence. Information from the NITT-MFP grantees suggests that the fellows have been a challenging population to reach following program completion. Respondents will not be penalized if they wish to skip questions or stop the interview.
Research suggests that monetary incentives to respondents have a strong positive effect on response rates and no known adverse effect on reliability. Research has shown improved response rates when remuneration is offered to respondents (Eyerman, Bowman, Butler, & Wright 2005).
Concern for privacy and protection of respondents’ rights will play a central part in the implementation of all study components. The NITT-MFP evaluation team members developing the instrument, conducting data collection, and analyzing the data have extensive experience protecting and maintaining the privacy of respondent data. All NITT-MFP data will be securely stored on a protected server. A SORN/PIA is currently being sought.
The contractor will obtain from each of the NITT-MFP grantees contact information for the fellows, including full name and e-mail address, to recruit the fellows for the NITT-MFP Interview. Randomly selected fellows will be contacted through e-mail about 4 to 5 months after completion of their fellowship to complete the NITT-MFP Fellow Interview. If the fellow agrees to participate, the NITT-MFP Fellow Interview will be completed by telephone. Respondents will be asked for their consent to record the interview. Recordings will be used to ensure that information is correctly captured and to correct and clarify brief written notes as needed and as part of data quality assurance procedures. Recordings will only be accessible to the contractor and will be stored on password-protected secure servers and destroyed once de-identified notes are completed. All contact information will be stored separately from other data collected and will be accessible only to select NITT-MFP evaluation staff.
The evaluation contractor’s NITT-MFP systems development team takes responsibility for ensuring that the data system is properly maintained and monitored. Server staff will follow standard procedures for applying security patches and conducting routine maintenance for system updates. Data will be stored on a password-protected server, and access to data in the system will be handled by a hierarchy of user roles, with each role conferring only the minimum access to system data needed to perform the necessary functions of the role.
For all data collection activities, the contractor will use passwords to safeguard all project directories and analysis files containing interview data to ensure that there is no inadvertent disclosure of study data. Contractor staff will also be trained on handling sensitive data and the importance of privacy. All contractor staff will sign a privacy pledge. In keeping with 45 CFR 46, Protection of Human Subjects, NITT-MFP procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent.
Data from the NITT-MFP interviews will be safeguarded in compliance with the Privacy Act of 1974 (5 U.S.C. 552a). The privacy of data records will be explained to all respondents during the consent process and in the consent forms.
No questions of a sensitive nature are being asked through the NITT-MFP Fellow Interview.
For the NITT-MFP Fellow Interview, a maximum of 66 fellows (10 fellows per 5 MFP-Y grantees and 8 fellows per 2 MFP-AC grantees) will be interviewed per year. The interview will be completed one time following fellowship completion. The interview is estimated to last one hour, including time for consent. The estimated burden time is based on the evaluation team’s expectations of the time required to conduct these interviews. Actual interview lengths may vary, but they will not exceed the burden estimate. There are no direct costs to respondents other than their time to complete the interview. Table 5 provides the number of annual respondents, number of responses, burden, and cost details for the fellow interview. Respondent costs are calculated as total burden hours × the average hourly wage for community and social service occupations (as reported in the 2014 Occupational Employment Statistics by the Bureau of Labor Statistics), which include occupations like substance abuse and behavioral disorder counselors, mental health counselors, mental health and substance abuse social workers, etc.
Table 5. Annualized Data Collection Burden
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage |
Total Respondent Costa |
NITT-MFP Fellow Interview |
66 |
1 |
66 |
1 |
66 |
$21.79 |
$1,438 |
a Total respondent cost is calculated as total burden hours x average hourly wage.
There are no respondent costs for capital or start-up or for operation or maintenance.
The annualized cost to the government is approximately $233,372. The estimated 5-year total cost to the government for the data collection is $1,166,860. This includes approximately $221,515 per year (or $1,107,575 total) for developing the instruments; managing and conducting data collection; respondent incentives; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $11,857 per year (or $59,285 total) represents SAMHSA costs to manage/administer the data collection and analysis for 10% of one employee (GS-14-4, $118,570 annual salary).
This is a new data collection.
Table 6 outlines the key time points for the study and for the collection of information.
Table 6. Time Schedule for Data Collection
Activity |
Time Schedule |
Prepare for data collection |
August–November 2015 |
Obtain OMB approval for data collection |
April 2016 |
Collect data |
April 2016–October 2019 |
Collect first round of NITT-MFP Fellow Interviews (Cohort 1) |
April–May 2016 |
Collect second round of NITT-MFP Fellow Interviews (Cohort 2) |
January–February 2017 |
Collect third round of NITT-MFP Fellow Interviews (Cohort 3) |
January–February 2018 |
Collect fourth round of NITT-MFP Fellow Interviews (Cohort 4) |
January–February 2019 |
Analyze data |
April 2016– November 2019 |
Disseminate
findings |
April 2016–February 2020 |
The NITT-MFP evaluation will help SAMHSA reach its diverse stakeholders through targeted products and innovative dissemination venues. The evaluation’s objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings. The dissemination plan includes products in a variety of formats for a variety of target audiences. Audiences for these reports will include Congress, SAMHSA Centers, the evaluation’s SAMHSA Contracting Officer’s Representatives (CORs), NITT-MFP grantees and participants, and the broader mental health and substance abuse fields (e.g., academia, researchers, policy-makers, providers). The NITT-MFP evaluation recognizes that different audiences are best reached by different types of report formats. For example, reports to Congress will require materials that are concise but offer policy-relevant recommendations. Reports created for SAMHSA Centers and the CORs will require more in-depth information, such as substantive background and discussion sections, to supplement the analytic approach. Reports created for NITT-MFP grantees will be concise handouts with helpful and easy-to-read graphics on performance data rather than lengthy text. The NITT-MFP evaluation will develop an assortment of disseminations products, including short and long analytic reports, congressional briefings, annual evaluation reports, research and policy briefs, ad hoc analytic reports, journal articles, best practice summaries, and conference or other presentations.
The process evaluation analyses will be conducted in each program year to provide ongoing information on the implementation and management of the fellowship programs and fellows’ experiences. Initial interview data will be used to develop themes or constructs that will be used to code the fellow interviews to identify commonalities and differences in their experiences using qualitative coding software.
The qualitative data drawn from the fellow interviews will provide evidence regarding any observed differences between grantees in terms of program outcomes. The analysis will include cross-grantee analyses that will examine shared and distinguishing strategies and process-outcome analyses, through comparisons of fellow outcomes by grantee, and examining relationships between the grantees and their program partners. The analysis will be guided by qualitative comparative analysis (QCA), a method designed for use with small sample and population sizes (Longest & Thoits, 2012; Rihoux & Ragin, 2008). QCA assists in the identification of cross-case patterns, using a comparative approach that respects the diversity of cases and their heterogeneity while identifying configurations of relevant conditions and contexts related to project outcomes.
OMB approval expiration dates will be displayed.
There are no exceptions to the certification statement. The certifications are included in this submission.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Snaauw, Roxanne |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |