SAMHSA Customer Satisfaction Survey
SAMHSA’s Performance Accountability and Reporting System (SPARS)
Training and Technical Assistance (TTA) Event Satisfaction
The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Performance Accountability and Reporting System (SPARS) contract includes provision of training and technical assistance (TTA) for SAMHSA discretionary services grantees and project officers on requirements, procedures, and strategies related to SPARS data collection, data entry, reports, and data use. Designed to improve the accuracy, completeness, and timeliness of grantee data, SPARS promotes the use of data to improve practice, program design, and policy development. SAMHSA is requesting approval from the Office of Management and Budget (OMB) to implement a satisfaction survey for a sample of participants of TTA events supported by the SPARS contract. SPARS considers it important to assess participants’ perception of the value of services provided in these events.
SPARS will use the satisfaction survey responses collected to guide decision making about the content and effectiveness of TTA events. The information gathered will help SPARS identify areas needing improvement and implement changes that are practical and meet participants’ needs. SPARS will use the knowledge and insight gained from the responses to plan and, if necessary, to redirect resources and efforts to improve or maintain a high quality of service to users of discretionary program performance data.
If SPARS does not collect this information, vital feedback regarding participants’ perception of the value of the TTA will be unavailable. This missed opportunity will hinder SPARS’s ability to develop, implement, and refine TTA materials and content in a manner best tailored for discretionary programs’ needs.
1. Data Collection Method
SPARS will collect data from a sample of participants at the completion of each TTA engagement or event using either a paper or a web-based survey. The web-based survey will be used in roughly 90 percent of events. Participants will have the option to decline responding to the survey. The SPARS team will use this survey for longer-term TTA events, not spot assistance, such as that provided through the SPARS Help Desk.
The survey will include seven (7) questions. The first question asks respondents to identify the TTA activity for which they are providing information; wherever feasible this will be pre-populated. The second question, which will also be pre-populated wherever possible, asks for the type of event to be identified (examples include online training, site visit or other on-site technical assistance, long-term telephone/email consultation, in-person conference presentation or workshop, annual or semi-annual grantee meeting presentation or workshop, or other event; in the case of “other,” a type can be specified by the respondent or pre-populated). The third question consists of a set of five (5) scoring elements for aspects of the TTA event, with Likert-type scales for agreement or disagreement. The next section of the survey asks about satisfaction with quality of instruction, quality of products and overall satisfaction, using a Likert-type scale. This is followed by an open-ended comment/recommendation question. The next section of the survey asks for a description of the respondents’ relationships to SAMHSA (grantees or staff, and their Center), and the grant(s) or grant program(s) with which they are associated.
Analyses will primarily consist of statistics to better understand the effectiveness and perceived value of the TTA provided.
2. Method for Identifying Respondents
SPARS asks all sampled participants in TTA events, including grantee leadership and SAMHSA Government Project Officers (GPOs), to provide feedback by completing a customer satisfaction survey.
Participation will be voluntary, and not completing the satisfaction survey will have no effect on course completion certification. Given the voluntary nature of the data collection efforts, SPARS will make efforts to achieve the highest response rate possible.
SPARS will ensure the protection of respondents’ identifying information, and identifying information will not be included in the analysis or reporting of responses.
There will be no requirements for respondents to maintain or save any data onto their local computers.
3. Proposed Sample Size and Rationale
Based on the contractor’s experience with similar projects, SPARS estimates that approximately 2,212 individuals per year will have occasion to complete a survey at the conclusion of a TTA event. This is based upon the current understanding of the user population (10,548), a desired level of overall precision of +/- 3.5 percent at a 95 percent level of confidence, and a response rate of 66 percent. This yields 1,106 sampled participants for each of an estimated two research intercepts annually.
4. Planned Frequency of Information Collection
SPARS will collect information upon completion of each TTA event, whether telephonic, web-based or in-person. SPARS will not conduct longitudinal follow-up (for example, to assess the longer-term impact of training). Current estimates are based on two events per SPARS user per year.
5. Methods for Identifying Duplications
The information collected through this survey is unique to this program and not available elsewhere.
6. Time Period Over Which Information Will Be Collected
Data collection will occur from clearance of the instrument through the end of the contract’s performance period in 2021.
7. Expected Response Rate and Plan for Follow-up for Nonrespondents
A response rate of 66 percent is expected. SPARS will send three rounds of prompts to nonrespondents; the original round will immediately follow webinars, modules and virtual trainings (including telephonic technical assistance), a reminder will be sent the following day, and a second reminder a week later. Follow ups for in-person events will occur three and five days following the events. Cross-referencing respondents against the universe of individuals to whom SPARS emailed the survey link will identify these nonrespondents. There will be no follow up for paper-based surveys.
8. Expected Ability to Assess Non-Response Bias Using Existing Information
A response rate of 66 percent is expected. SPARS does not expect nonresponse bias, but any knowledge of characteristics of nonrespondents based on pre-sample information (where present) will allow us to determine whether nonrespondents are distinct with regard to program experience, program type, or Center.
9. Methods Used to Maintain Customer Privacy
SPARS will submit the results of individual TA surveys to SAMHSA in aggregate, for example, via monthly reports. In addition, SPARS will not report individual respondents’ organizational affiliations to SAMHSA. In order to protect the responses for programs or Centers with only a single respondent, analytical results for programs or Centers with only a single potential respondent will be combined with those for other similar programs within the same Center; those that cannot be aggregated in this manner will have their individual program-specific results suppressed in reporting (with explanatory footnotes included in tables), but will be combined with other programs in the same Center, and with those with similar activity profile data using survey responses. In the same vein, SPARS may either aggregate all Center responses together or suppress the reporting output of results where presentation of data would allow a Center’s sole respondent’s answers to be identified. Any subsequent discussion of analysis “by program” should be understood to include such combinations or suppression of small-cell results.
Training/TA Provider |
Number of Respondents |
Responses per Respondent (Annual) |
Total Number of Responses (Annual) |
Hours/ Response |
Total Hours |
Hourly Wage1 |
Total Hourly Cost |
|
|||||||
SPARS |
1,106 |
2 |
2,212 |
0.1 |
221 |
$20.64 |
$4,561 |
1 Hourly Wage: The $20.64 rate is used as the mean hourly wage as reported on the Bureau of Labor Statistics website (http://www.bls.gov/) for substance abuse an behavioral disorder counselors. Annual hours are based on a 40-hour workweek for 48 weeks per year.
The total annualized burden to an estimated 2,212 sampled respondents for SPARS TTA satisfaction data collection is estimated to be 221 hours. Burden estimates are based on cognitive/pilot testing of instruments by the SPARS TTA contractors. The annualized hourly costs to sampled respondents are estimated to be $4,561. Hourly wage information is based on estimated median hourly wages of $20.64 an hour for substance abuse and behavioral disorder counselors as reported in the Occupational Employment Statistics available from the Bureau of Labor Statistics, U.S. Department of Labor (http://www.bls.gov/oes/current/oes211011.htm). There are no direct costs to respondents for participation aside from their time. For the purposes of estimating annual cost, SPARS assumes that participants will participate in two events per year.
The annual estimated cost to the government for the SPARS program is $10.6 million. Approximately $78,234 per year represents SAMHSA costs to manage/administrate the program for 90% of one employee (GS-13).
SPARS developed the survey questions based on standard instructional design and assessment practices in accordance with Kirkpatrick’s four-level evaluation model. The instrument measures elements of self-reported satisfaction with events. Respondents are asked to directly report on their own experiences with these elements of the events, which should minimize any interpretive or construct-validity challenges. Timing for burden estimates and question wording are based on cognitive testing of survey questions with SPARS TA center staff, and discussions with SAMHSA staff, as well as the combined decades of experience of SPARS TA center managers with TA project monitoring and evaluation.
The common measures submitted here for OMB approval are the result of lengthy consultation and discussion among SAMHSA/SPARS personnel and training staff. SAMHSA/SPARS senior officials made the final selection of these measures.
The consultant within SAMHSA for the survey is Dr. Darren Fulmore, who also serves as the Contracting Reporting Officer (COR) for the SPARS contract. Dr. Fulmore’s contact information is
Darren Fulmore, Ph.D.
SAMHSA’s Performance Accountability Reporting System (SPARS)
Contracting Reporting Officer
SAMHSA
5600 Fishers Lane, 15E33-D
Rockville, MD 20852
(301) 945-3214
darren.fulmore@samhsa.hhs.gov
Our statistical consultant for the design is Dr. Steven Sullivan of Cloudburst Consulting Group, Inc. Dr. Sullivan is an econometrician with a history of successful study design and implementation for SAMHSA, including the data collection and evaluation planning for the Co-Occurring Disorders Integration and Innovation (CODI) contract and the Grantee Data Technical Assistance (GDTA) contract. He is also the lead for the data collection efforts and the analysis of satisfaction data from the survey. Dr. Sullivan’s contact information is:
Steven T. Sullivan, Ph.D.
Senior Director and Subject Matter Expert
Cloudburst Consulting Group, Inc.
8400 Corporate Drive, Suite 550
Landover, MD 20785
(301) 385-6693
steven.sullivan@cloudburstgroup.com
Attachment A: SPARS Satisfaction Survey
Attachment B: Email to respondents
Attachment C: Screenshots of Survey
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | RMC Basic Report & Proposal Template |
| Author | RMC Research Corporation |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |