Pre-Test of Instruments of Psychosocial Care for the Treatment of Adults with PTSD
Office of Management and Budget Supporting Statement
Part B: Collection of Information Employing
Statistical Methods
October 29, 2013
Submitted by:
Office of the
Assistant Secretary for Planning and Evaluation
Office of Disability, Aging and Long-Term Care Policy
U.S. Department of Health and Human Services
Hubert H. Humphrey Building, Room 4E
200 Independence Ave, SW
Washington, DC 20201
Project Officer:
Kirsten Beronio
CONTENTS
B. Statistical Methods(used for collection of information employing statistical methods)
1. Respondent Universe and Sampling Methods 1
2. Procedures for the Collection of Information 2
3. Methods to Maximize Response Rates and Deal with Nonresponse 6
4. Tests of Procedures or Methods to Be Undertaken 7
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 7
B. Collection of Information Employing Statistical Methods
ASPE’s contractor, Mathematica Policy Research, will partner with six behavioral health organizations that provide treatment to adults with post-traumatic stress disorder (PTSD) to pre-test surveys (for clinicians, clinician supervisors, and consumers) of quality of psychotherapy for adults with PTSD in concordance with evidence-based strategies. Mathematica will utilize its existing relationships with behavioral health organizations to identify partner organizations willing to participate in the testing that are representative of the types of providers that deliver PTSD treatment. These may include hospitals, community-based organizations, university-affiliated clinics, and Veterans’ Affairs clinics/hospitals. Potential sites will be sent an outreach email with a brief recruitment document containing project and data collection information (Attachment 10) and a project description (Attachment 11). Follow-up phone calls will be made to interested sites to determine their suitability for data collection (Attachment 4). Mathematica will select organizations with ample respondents—clinicians treating adults with PTSD, supervisors, and consumers—to meet the sampling needs described below. Mathematica will also give preference to sites where clinicians have a range of educational backgrounds (M.S.W., M.D., Ph.D.) and therapeutic orientations (cognitive behavioral, psychoanalytic), and where clinicians are regularly supervised. The main criterion in site selection will be whether there are enough clinicians, supervisors, and consumers to serve the pre-testing needs. Mathematica anticipates that the final six sites will be selected within two months after receiving clearance.
Four survey respondent types will be involved in pre-testing the survey instruments: clinicians, clinician supervisors, consumers, and site coordinators. Below we describe the sampling method for each respondent type.
Clinicians. The initial sampling universe will consist of clinicians who currently provide psychotherapy to at least three adults with PTSD in the participating behavioral health organizations. The site data collection coordinators within each of the organizations will develop a list of eligible clinicians, their consumers (i.e., clients), and the consumers’ length of time in therapy for PTSD. This list will be developed utilizing the Site Coordinator’s Sample Section Abstraction Form (Attachment 5). A purposive sample of 36 clinicians with a higher number of consumers in treatment for PTSD will be selected. Additionally, across the six behavioral health organizations, Mathematica will sample clinicians with varying academic backgrounds and therapeutic orientations. For example, one site (site A) may have a large number of clinicians who have a Ph.D. and provide cognitive behavioral therapy (CBT). Another site (site B) may have a 50-50 mixture of Ph.D.- and M.S.W.-level clinicians who provide CBT and psychodynamic therapy. In site A, Mathematica would sample the Ph.D. clinicians providing CBT who had the highest number of PTSD patients. They would then sample fewer Ph.D.-level clinicians from site B and instead favor the M.S.W.-level clinicians providing either CBT or psychodynamic therapy. The sampled clinicians will complete the survey a total of three times, following therapy sessions provided to three different consumers. A sampling frame based on the clinicians’ client list will be developed to select the therapy sessions about which the clinicians will complete the survey. The sampling frame will group consumers into those who have recently begun treatment, those in the middle of treatment, and those toward the end of treatment.1 From our sampled clinicians, one consumer in each stage of treatment will be randomly selected.
Prior to completing their first survey, clinicians and supervisors will receive an email with instructions on how to create a user account to access the survey. This email will include the survey URL, a unique user name, and a temporary password for first log-in. Account creation will involve verification of the respondent’s name and email address, and completion of a brief one-time demographics questionnaire (Attachment 6). After the therapy session, the clinician will then complete the survey for each selected consumer based on the care provided to that consumer in the therapy session.
Clinical supervisors. Clinical supervisors for each sampled clinician will be included in the data collection effort. In the Site Coordinator’s Sample Section Abstraction Form (Attachment 5), the clinician’s supervisor will also be listed. In general, supervisors provide oversight for multiple clinicians. We conservatively estimate, therefore, that six clinical supervisors (i.e., one supervisor per behavioral health organization will participate. Each supervisor will complete the survey on care delivered in three separate sessions, by each supervisee, for a total of 18 completed surveys per supervisor. The clinical supervisors and the clinicians will complete the survey on care delivered in the same therapy session
Consumers. The universe of consumer respondents is individuals who receive psychotherapy for PTSD from a sampled clinician and for whom the clinician is completing the survey. As described above, 36 clinicians will complete the survey on 108 consumers; those 108 consumers will also complete the survey.
Site coordinators. A staff member from each of the six sites will participate in a one-hour phone call with Mathematica to collect basic information about the site (as listed in Attachment 4, the Site Coordinator Checklist). They will also participate in “check-in” phone calls with Mathematica to discuss successes and challenges with data collection. Six coordinators will complete these phone calls.
Information will be collected from the six sites via a one-hour telephone call with Mathematica staff (using Attachment 4, Site Coordinator Checklist) in order to determine site eligibility for data collection. We anticipate a 100 percent response rate, since only interested organizations that respond to our recruitment email will be contacted.
Data collection will be carried out using the three web-based surveys mentioned above—one to be completed by clinicians, another by clinical supervisors, and a third by consumers. Clinicians and clinical supervisors will also complete an additional brief questionnaire (Attachment 6) to gather information on their demographic characteristics.
We estimate that 100 percent of clinicians will complete the questionnaire on their demographic characteristics as it is a requirement of participation in pre-testing activities; clinicians will not be able to access the online survey until the demographics questionnaire has been completed.
We conservatively estimate that 50 percent of clinicians will complete the survey (Attachment 1).2 As described previously, each clinician will complete the survey on three consumers following each consumer’s therapy session. Twenty-four hours prior to a given session, site coordinators will notify clinicians by email of the selected consumer (Attachment 12). This email will be a generic reminder containing no personal identifiable information. It will contain a site-generated unique identifier (e.g., patient ID number), date and time of session, and website link.
When a survey is not completed on the care delivered to a sampled consumer in a sampled therapy session, Mathematica staff and the site coordinator will communicate with one another to determine if the survey should be completed on care delivered to the same consumer at the next attended session or if the survey should be completed on a newly sampled consumer. For example, if the clinician forgot to complete the survey for a sampled session and the consumer has now moved into a different phase of treatment (i.e., from an early phase to a later phase of treatment) or if the consumer has terminated treatment, Mathematica will select a new individual in the same phase of treatment from the clinician’s client list. The site coordinator will be notified of this selection and will in turn notify the clinician of the new consumer and session selection. If the consumer missed the session and therefore will remain in the same session phase at his or her next appointment, the clinician will complete the survey upon their next session. Mathematica staff will again notify site coordinators, who will in turn notify clinicians of the next survey completion date. For each new appointment, clinicians will again receive a reminder email.
We also conservatively estimate that 50 percent of consumers will complete the survey.3 Mathematica will notify the site coordinators of the selected consumers and their session date. Depending on how sites manage their sessions, sites will introduce the study and provide consumers with the study description document and consent statement (Attachment 9) either at check-in for the current session or after the session is completed. Consumers will also be provided with the URL for their secure survey and instructed to complete the survey online. For consumers without Internet access, paper-based copies with prepaid return envelopes will be provided. If the consumer discontinues treatment, another consumer being treated by the same clinician and in the same stage of treatment (i.e., early, middle, end) will be randomly chosen using the same procedure described above. If the consumer misses the intended session, the procedures described above will be followed again. The number of consumer substitutions will be recorded for response rate purposes.
We assume that each participating behavioral health organization will have one clinical supervisor, for a total of six participating clinical supervisors. If a site has more than one clinical supervisor, each participating clinician’s supervisor will be contacted to complete the demographics questionnaire and survey. We expect 100 percent of the clinical supervisors to complete demographics questionnaire, because its completion is a condition of participation in the pre-testing activities and accessing the online survey.
Supervisors will be contacted through the same procedures used to remind the clinicians to complete the survey—that is, via email from the site coordinator. When a new consumer is selected because of a missed session or discontinued treatment, the supervisor will also be notified in the same manner as clinicians. The clinical supervisors will complete the web-based survey after reviewing videotapes or audiotapes of the clinician’s sessions (if their site routinely tapes clinicians providing care as part of the clinician oversight and quality-control process), or after live supervision through one-way glass.4 Because the organizations recruited to pre-test the surveys will provide supervision as part of their standard practice, we expect a 93 percent response rate,5 yielding 108 surveys completed by clinical supervisors. Table B.1 illustrates our response rate assumptions.
Table B.1. Response Rate Assumptions
Task |
Number of Respondents |
Response Rate (%) |
Total |
Site Coordinator Checklist |
6 |
100 |
6 |
Demographics Questionnaire |
|
|
|
Respondent Type |
|
|
|
Clinician |
36 |
100 |
36 |
Clinical Supervisor |
6 |
100 |
6 |
Survey |
Sampled Sessions |
Response Rate (%) |
Total |
Respondent Type |
|
|
|
Clinician |
216 |
50 |
108 |
Clinical Supervisor |
116 |
93 |
108 |
Consumer
|
216 |
50 |
108 |
In this data collection effort, statistical methodologies will be used to (1) examine the factor structure of the clinician, clinician supervisor, and consumer surveys; (2) conduct a preliminary assessment of the surveys’ reliability; and (3) conduct a preliminary assessment of the surveys’ validity. These preliminary, exploratory analyses will be used to inform approaches to scoring, item retention, and improvement of each survey. The sample sizes we have described will be sufficient to support our analytic approach. Below we describe the minimum sample sizes required to conduct each of these analyses.
Sample size estimates for factor analyses. There are two dominant approaches to calculating the sample size for factor-analytic and structural equation models. The first approach uses a linear heuristic, such as the “rule of 5” or the “rule of 10” (Hatcher 1994; Nunnally 1978). These rules suggest that the sample size should be at least 5 or 10 times the number of indicators. While this approach offers a simple way to calculate the number of respondents needed, it assumes that sample size is a linear function of the number of indicators.
An alternate approach is to calculate sample size based upon the characteristics of the factor-analytic model to be estimated. This approach assumes that sample size is a nonlinear function of the number of potential combinations of latent variables, cross-loadings of indicators, and correlations between the error variances. Table B.2 presents the necessary sample sizes to conduct factor analyses using this approach. The sample size estimates were calculated as a function of the models’ complexity (degrees of freedom6) and according to the ability to detect a difference between the null and alternative hypotheses of the models’ goodness of fit to the data using the root mean square error of approximation (RMSEA) statistic, which is often used in factor-analytic models (Cramer 2003; MacCallum et al. 1996). RMSEA values below 0.08 suggest a good fit, while values between 0.08 and 1.00 generally suggest a marginal fit (Fabrigar et al. 1999). In the estimates below, the null hypothesis RMSEA was set at 0.08. Based upon the parameters described in Table B.2, 69 clinician responses and 103 consumer responses are needed to detect a factor-analytic model with a RMSEA of 0.06 (i.e., a 0.02 difference between the 0.08 threshold and a RMSEA value of 0.06).
Table B.2. Required Sample Size for the Clinician, Supervisor, and Consumer Factor-Analytic Models
Difference Between Null and Alternate RMSEA |
Sample Size for Clinician and Supervisor Model |
Sample Size for Consumer Model |
.05 |
31 |
43 |
.04 |
36 |
51 |
.03 |
46 |
66 |
.02 |
69 |
103 |
.01 |
165 |
269 |
Note: Based upon the number of survey items, the clinician and consumer models assume 36 indicators and 27 indicators, respectively, and five underlying constructs. All calculations assume a 95 percent confidence level, 80 percent power, and a 5 percent Type I error. The number of degrees of freedom might change depending on specification of the model, i.e., change in the number of latent factors, cross-loadings of the items, and correlation between residual variances, etc.
Sample size for preliminary assessments of reliability and validity. In addition to calculating the sample size for the factor-analytic models, we also estimated the sample size required for computing preliminary assessments of reliability and validity. We will first provide basic descriptive statistics on the responses from each survey (i.e., means, ranges, percentages of survey items skipped or not answered). We plan to estimate inter-rater reliability (clinician rating versus the supervisor rating of the same session, and the clinician rating versus the consumer rating of the same session) by calculating kappa. We set the level of kappa at 0.80 (with a confidence interval range from 0.65 to 0.95) and the proportion of positive ratings of raters at 0.20. In order to meet these rather conservative assumptions, a sample size of 98 respondents is required.
To preliminarily assess the surveys’ validity, we will calculate sensitivity and specificity. For both the clinician and consumer surveys, we will use the clinical supervisors’ ratings as the gold standard. To calculate the sample size needed to conduct these analyses, we set the levels of sensitivity, specificity, and the expected prevalence at the 0.50 level, the desired precision at 0.15, the level of power at 0.80, and the alpha level at 0.05. The sample size required to meet these assumptions is 87 respondents for each respondent type (clinician, supervisor, and consumers).
Sample size summary. For the consumer survey, a minimum sample size of 114 respondents is required to satisfy the assumptions for both the factor-analytic model and the inter-rater reliability and sensitivity-specificity analyses at the .80 power level and .05 alpha. For the clinician survey, a minimum sample size of 98 respondents is necessary to satisfy the assumptions for the factor-analytic model and the inter-relater reliability and sensitivity-specificity analyses.
A team of experienced quantitative researchers will analyze the survey data collected as part of this pre-test. In addition, statisticians will review (1) specifications for the coding that will be used to conduct the analyses, and (2) descriptions of the results to ensure that appropriate conclusions are drawn. A senior programmer will review all code developed for the analyses.
B3. Methods to Maximize Response Rates and Deal with Nonresponse
In selecting behavioral health organizations to participate in this pre-testing effort, we will seek organizations whose clinicians and clinical supervisors are interested in participating and able to participate. This approach should reduce nonresponse. We expect to find some interest in participating on the part of organizations such as hospitals, which routinely engage in quality-of-care activities, and which may have a vested interest in and see the benefits of developing instruments for assessing the quality of care clinicians provide for treatment of PTSD. Nonetheless, behavioral health clinicians and consumers can be a challenging population to survey because individuals may discontinue therapy and have competing demands for their time. We estimate a 50 percent response rate and will sample 216 consumers to participate to yield 108 responses for clinicians and consumers.
The relevance of the survey topic (assessing the delivery of evidence-based care), and the possibility that the survey could ultimately help to improve the quality of PTSD care, may encourage clinicians and consumers to respond. In any case, numerous methods and materials will be used to encourage response and reduce challenges to participation:
Training. Mathematica will train the site coordinators at each site in data collection procedures, including techniques for minimizing nonresponse.
Support. Mathematica will meet regularly with the site coordinators to proactively discuss—and identify solutions to—issues of nonresponse, and to review data collection successes and challenges.
Reminders. The site coordinators will routinely prompt the participating clinicians and clinical supervisors to complete the survey and will solicit their feedback regarding any challenges encountered in completing the survey so that solutions can be identified.
Informational materials. During the data collection period, site coordinators will provide consumers with informational materials about the project and in this way let them know that they may be invited to participate. Site coordinators will also explain the study to consumers and will emphasize its importance, confidentiality of the consumers’ answers, and the $20 incentive (described below).
Closed-ended questions. To facilitate completion of the survey, the survey items are closed-ended with categorical response categories.
Brief survey. To facilitate completion of the survey, it is designed to be completed in 5 to 10 minutes.
Use of incentives for consumers. Consumers will receive a $20 gift card in recognition of their participation; this is intended to increase the likelihood that they will complete the survey.
Clinicians, consumers, and a technical expert panel have already provided ASPE’s contractor with feedback on the wording of the survey items. During the data collection effort, information will be gathered to inform decisions regarding the final content of the surveys, the optimal timing of survey administration, and additional testing needed to fully develop the surveys. This information will be gathered in the following ways:
Regular meetings with the site coordinators. Mathematica will meet with site coordinators on a regular basis and will include a debrief in our final meeting. This process will allow us to document challenges and barriers associated with data collection, as well as the solutions we devised to any issues the site coordinators identify. This information will help us form best practices for survey administration in the various settings.
Examination of site and clinician characteristics. Mathematica will examine the characteristics of both the sites and the clinicians in terms of the overall responses from the survey (i.e., item responses) and the response rate. This step will allow a further examination of the feasibility of implementing the measures among varying clinician types and organizations.
Examination of survey paradata (data on survey data collection). These data will automatically be collected as part of the web survey administration and through regular conversations with site coordinators to discuss data collection successes and challenges. We will examine the three surveys’ paradata to inform overall survey administration. Examining total time spent completing the survey and question-response time will help us ensure the survey is administered in a time-efficient manner. Examining the date of completion and comparing it to the date the session occurred can, in conjunction with data on administration and question-response time, help assess the degree to which retrospective recall may be impaired.
Examination of survey response data. The response data from each of the surveys will be used to conduct the analyses mentioned above (i.e., factor analysis, reliability, and validity). The results from these analyses will allow us to optimize the survey items.
ASPE, in partnership with the National Institute of Mental Health (NIMH), has contracted with Mathematica and its contractor, the National Committee for Quality Assurance (NCQA), to develop and pre-test the surveys. Table B.3 identifies the individuals at these organizations who were consulted regarding the quantitative methods used in this project.
Table B.3. Individuals Consulted About or Involved in Data Collection and Statistical Analysis Plans
Name |
Title/Organization |
Kirsten Beronio, J.D. |
Director, Division of Behavioral Health and Intellectual Disabilities Policy, Office of Disability, Aging, and Long-Term Care Policy (DALTCP), ASPE; contract officer representative (COR) for this project |
D.E.B. Potter, M.S. |
Senior survey statistician, Agency for Healthcare Research and Quality (AHRQ) and DALTCP ASPE (on detail from AHRQ to ASPE two days a week); Deputy COR for this project |
Joel Dubenitz, Ph.D.; licensed psychologist |
Social science analyst, DALTCP ASPE |
Joel Sherrill, Ph.D. |
Program chief, Psychosocial Treatment Research Program, Division of Services and Intervention Research, NIMH, National Institutes of Health |
Kirsten Barrett, Ph.D. |
Senior survey researcher, Mathematica |
Frank Yoon, Ph.D. |
Senior statistician, Mathematica |
Melissa Azur, Ph.D. |
Senior researcher, Mathematica |
Daniel Friend, M.S. |
Survey researcher, Mathematica |
Dmitriy Poznyak, Ph.D. |
Statistician, Mathematica |
Robert Saunders |
Assistant vice president, research and analysis, NCQA |
Peichang Shi |
Senior health care analyst, NCQA |
Attachments
Clinician’s Survey of the Delivery of Evidence-Based Psychotherapy
Clinician Supervisor’s Survey of the Delivery of Evidence -Based Psychotherapy
Consumer’s Survey of the Delivery of Evidence-Based Psychotherapy
Site Coordinator’s Survey to Obtain Site-of-Care Characteristics
Site Coordinator’s Sample Section Abstraction Form Used to Identify Persons Being Treated for PTSD and Their Associated PTSD Clinician and Clinician Supervisor
Clinician and Supervisor Demographics Questionnaire
Section 301 of the Public Health Service Act [42 U.S.C. 241]
60-Day Federal Register Notice
Project Background/Invitation to Participate, and Consumer Participation Consent Forms
Site Recruitment Document
Project Description
Site Coordinator Reminder to Clinicians
Supporting Statement A References
Supporting Statement B References
1 The expected number of therapy sessions provided to adults with PTSD varies from organization to organization. Mathematica will work with each organization to define the “early,” “middle,” and “end” phases of treatment. For example, the recommended number of sessions for cognitive processing therapy, a type of cognitive behavioral therapy, is over 12 (Resick and Schnicke 1993). If a site (or clinician) uses this form of therapy, treatment phases would be understood as follows: (1) early = 2–4 sessions, (2) middle = 5–8 sessions; (3) late = 9–11 sessions. Our sampling frame will not include the first and last sessions, as these are generally intake/assessment and debrief sessions.
2 Provider response rates have ranged from 75 percent with military mental health providers to 33 percent with practitioners in private clinics. We conservatively estimate a 50 percent response rate due to the variety of sites we wish to recruit.
3 This estimate is based on a recent survey from the Perceptions of Electronic Health Records and Their Effect on the Quality of Care project, funded by the U.S. Department of Health and Human Services, which achieved a 53 percent response rate among 670 consumers surveyed immediately after a provider visit.
4 Only behavioral health organizations that provide clinical supervision as part of standard operations will be invited to participate in this data collection effort. Thus the organizations will already have processes in place to review videotapes or audiotapes or to view live sessions.
5This estimate is based on results from a recent survey with providers (Quality Reviews of the Mental Health Residential Rehabilitation Treatment Programs Project, funded by the U.S. Department of Veterans Affairs), where a 100 percent response rate was achieved due to high buy-in with the organization. We are assuming a more conservative response rate as the study funded by the U.S. Department of Veterans Affairs was based on a single-payer environment.
6 We calculated the models’ degrees of freedom using the following formula: DF = m*(m+1)/2-2*m-ξ*(ξ-1)/2. The first term, m*(m+1)/2, represents the total number of elements in the variance-covariance matrix to be analyzed; that is the total available number of degrees of freedom. The second term, 2*m, represents the number of parameters to be estimated in the matrix of loadings, the variance-covariance matrix of measurement error terms, and the variance of constructs. Finally, the third term, ξ*(ξ-1)/2, represents the free off-diagonal covariances of the constructs.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Melissa Azur |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |