NCTSI OMB Renewal Statement_032009

The Center for Mental Health Services (CMHS), Substance Abuse and Mental Health Services Administration (SAMHSA), is requesting a revision from OMB to approve data collection associated with the cross-site evaluation of the National Child Traumatic Stress Initiative (NCTSI). The current approval is under OMB No. 0930 0276 which expires on 4/30/2009.The overall presence of traumatic events in life is high in the United States, for children and for adults, in general, and especially for adults who experienced traumas in childhood, as findings from the National Comorbidity Survey (NCS) and other surveys and studies indicate. Conducted between 1990 and 1992, the NCS found that the lifetime prevalence of experiencing a traumatic event severe enough to cause psychopathology, such as posttraumatic stress disorder (PTSD), is more than 50%. Approximately 20% to 25% of individuals who experience a traumatic event go on to develop PTSD, with the lifetime prevalence of PTSD estimated to be nearly 8% in the general population. Qualifying events for PTSD and other trauma-related disorders were common, with many respondents reporting the several such events during their lifetimes (Kessler, Sonnega, Bromet, Hughes, & Nelson, 1995). Moreover, many of the traumas reported by adults in the survey in this sample of Americans aged 15–54 were actually experienced in childhood.

Children’s experience of trauma and trauma-related disorders occurs as a result of child maltreatment; witnessing or experiencing community or domestic violence; accidents; injury; terrorist acts or events due to war; witnessing or experiencing natural disasters, such as floods, hurricanes, and fires; experiencing loss of family or friends; displacement and refugee trauma; and medical trauma—all of which can have devastating effects on children and their families. The current climate of war and heightened risk of terrorism in the United States only increase the potential that children will experience trauma as a result. Even for those who have not experienced war and terrorism firsthand, the media has made such events readily accessible. A number of recent studies have reported an association between televised violence and experience of diagnosable PTSD (Brener, Simon, Anderson, Barrios, & Small, 2002; Ozmert, Toyran, & Yurdakok, 2002; Pfefferbaum, 1999; Terr et al., 1999). Recent research continues to document that trauma often leads to a wide range of psychopathologies capable of having lifelong effects and intergenerational impact (Breslau, Davis, Andreski, & Peterson, 1991; Hubbard, Realmuto, Northwood, & Masten, 1995; National Advisory Mental Health Council, 2001).

Beyond violence in the local and global community, many children incur just as much risk, if not more, of traumatic experience in their own homes or at the hand of someone close to them. In 2002, an estimated 896,000 children were victims of child abuse or neglect in the United States, and more than 60% were neglected by their parents or other caregivers (U.S. Department of Health and Human Services, 2002). For those children surviving child maltreatment, the negative impact on their psychological health can be significant and enduring. Studies of children in violent homes have shown increased dissociation and other trauma-related symptoms relative to children in nonviolent homes (Rossman, 1999), as well as lower self-esteem, lower levels of social functioning, and higher levels of depression, anxiety, behavioral problems, and aggressive behavior (Fantuzzo et al., 1991; Graham-Bermann, 1998). Long-term impairments in adulthood have included sexual disturbances, anxiety and fear, depression, low self-esteem, aggressive behavior, PTSD, and interpersonal problems (Silverman, Reinherz, & Gianconia, 1996).

Many studies have documented the long-term negative effect on children of a range of different trauma exposures (Honig, Grace, Lindy, Newman, & Titchener, 1993; Hubbard et al., 1995; Išpanovic-Radojkovic, 1993; Jones & Ribbe, 1991; Widom, 1989). Decades after her landmark child trauma study, researcher Lenore Terr described how childhood psychic trauma appears to be a crucial etiological factor in the development of a number of serious disorders both in childhood and adulthood (Terr et al., 1999). Davis (2000), who comprehensively reviewed the literature, noted that PTSD appears to be a potentially serious disorder in children and adolescents, not only because of the intense suffering it wreaks on young people, but because of its adverse effect on biological, psychological, and social development. Additionally, Tyano and others (1996) found that in the case of more severe PTSD symptoms, children were more likely to have longer term maladjustment.

Research has shown that appropriate intervention at the appropriate time can drastically affect whether and to what extent children recover from exposure to trauma. For example, counseling children very soon after a catastrophic event has been shown to reduce some symptomatology (Chemtob, Nakashima, & Carlson, 2002; Goenjian, Karayan, & Pynoos, 1997). Studies also have shown that because a parent’s reaction to the event strongly influences children’s ability to recover, children are more likely to suffer fewer effects when parents receive emotional support or counseling following a traumatic stressor (Cohen, Mannarino, & Knudsen, 2004; De Clercq, 1995; Keren & Tyano, 2000). Although the evidence base for treatment and intervention is growing, a comprehensive report from the National Advisory Mental Health Council’s Behavioral Science Workgroup indicates that even in the case of treatments found to be effective, the protocols are not widely understood by clinicians in the field, and are not being translated into practice often enough (National Advisory Mental Health Council, 2001).

The National Child Traumatic Stress Network

As research and recent national reports have suggested, without a coordinated and sustained effort to address the gaps in children’s mental health research and treatment, many children will miss an opportunity for care and recovery from traumatic experiences, as well as a chance “to live, work, learn, and participate fully in their communities” (New Freedom Commission on Mental Health, 2003, p. 1). In building a bridge between science and services—and between services and future research—the NCTSI has the potential to simultaneously fulfill many priority needs identified by a consensus of experts, including the need for implementation of evidence-based interventions and for research studies to inform the development of refined and increasingly effective treatment interventions.

Organized into three interlinking tiers, the National Child Traumatic Stress Network (NCTSN) comprises a nationwide network of over 80 current and previously funded grantees involved in diverse ways with improving access to care and raising the standard of care for children exposed to trauma. The current grantees include 37 Community Treatment and Services Programs (CTSs), 13 Treatment and Service Adaptation Centers (TSAs), and 2 grantees that together compose the National Center for Child Traumatic Stress (NCCTS). The CTSs provide services in community settings, collect clinical data on traumatized children receiving treatment, provide leadership and training on child trauma for service providers in the community, and in some instances serve as community “laboratories” for effectiveness studies of trauma-specific interventions and treatments (NCTSN, 2008). The TSAs identify, support, improve, and develop treatment and service approaches for different types of child and adolescent traumatic events (NCTSN, 2008). TSAs also may provide direct services to children who experienced traumatic events and their families, but they are usually involved in the development and testing of trauma-specific interventions and treatments, as well as the dissemination of best practices to mental health professionals. TSAs funded before 2005 were called Intervention and Development Evaluation Centers (IDEs), but throughout this statement, the term TSA will be used to refer to both TSAs and IDEs. The NCCTS provides leadership and guidance in coordinating activities to the network of grantees. In addition, the NCCTS develops and disseminates evaluation, treatment, and public mental health strategies for children, families, and communities affected by traumatic events (NCTSN, 2008). Through its multitiered structure, the NCTSN draws on the strengths of academic centers, hospitals, community-based agencies, schools, and other entities to respond to children’s immediate clinical needs, while establishing lasting partnerships and relationships needed to lay the foundation for a network of services organized along a continuum of care.

The characteristics of individual NCTSN grantees reflect attention to national priorities regarding the quality and type of services and research needed in the field. Grantees serve geographically, demographically, and clinically diverse populations and provide a range of treatments, addressing the need for a broad array of services to meet eclectic needs. Moreover, many grantees have responded to the call for building interagency relationships among entities serving children and adolescents at the local level. Many grantees are working to involve families in the planning and delivery of their services. Some are particularly focused on the development and evaluation of interventions that are developmentally appropriate, and others evaluate and disseminate interventions that respond to the needs of culturally diverse populations.

The Need for Evaluation

In 2004, SAMHSA issued a request for task order proposals (RFTP) for the cross-site evaluation, which had the following objectives:

Assess the evaluation activities conducted by grantee sites
Report evaluation results and lessons learned on the basis of a cross-site analysis of existing evaluation data
Develop logic models that delineate the relationship between center collaboration and improved outcomes for children who have experienced traumatic stress
Develop a comprehensive data collection package
Coordinate with the NCCTS to ensure cross-site evaluation activities build on the work of grantee sites
Create materials and training plans to assist grantees in meeting future evaluation requirements

The issuance of this RFTP can be attributed to several factors, including the initial congressional evaluation requirements for the NCTSI in 2000 (P.L. 106–310), goals and recommendations stemming from the 2003 report from the President’s New Freedom Commission on Mental Health (specifically, Goal 5), and Federal program requirements as outlined within the Government Performance and Results Act of 1993 (GPRA), and the Office of Management and Budget’s (OMB’s) Program Assessment Rating Tool (PART). PART requirements for the NCTSI include regularly scheduled, objective, independent evaluations that examine how well the program is accomplishing its mission and meeting its long-term goals. These evaluations should be conducted by nonbiased parties with no conflict of interest and should include recommendations for improving program performance.

The evaluation has been and will continue to be focused on the organization, collaborative efforts, function, and impacts of the NCTSI as a whole rather than designed to assess the effectiveness of specific programs or interventions. Cross-site evaluation data will be used to

determine the extent to which the NCTSI, through the NCTSN, has been able to achieve its goal of improving mental health services and access to care for children and adolescents, while improving the evidence base;
assist the NCTSN better meet its goals;
focus technical assistance and support; and
ensure accountability to stakeholders, including Federal agencies and the children and families served by the NCTSN, by informing them of progress made by the NCTSN nationwide.

Evaluation data provide the information necessary for shaping and influencing program and policy development through the systematic analysis and aggregation of information across the components of large-scale initiatives, thus contributing to an understanding of overall program effectiveness. Moreover, as challenging as evaluation of large-scale multisite initiatives like the NCTSI can be, without comprehensive evaluation information, the implementation of programs cannot be monitored effectively and their expected outcomes and large-scale product dissemination may be difficult to identify.

Previously Approved Clearance

The previously submitted OMB clearance request was approved for the first 3 years of the cross-site evaluation of the NCTSI.

The goals of the cross-site evaluation are to describe the children and families served by the NCTSN and their outcomes, assess the development and dissemination of effective treatments and services, assess intra-Network collaboration, and assess the Network’s impact beyond the NCTSN. The evaluation addresses the following overarching research questions in order to attain these goals:

Who are the families and children being served by the NCTSN centers, and to what extent do their outcomes improve over time?
What type and amount of services do children and families receive?
How satisfied are children and families with the services they have received?
What impact has the NCTSN had on affiliated providers’ knowledge and practice of trauma-informed services?
What products/innovations have been developed and disseminated within the Network, and what factors influence product/innovation development and dissemination?
What Network-generated products/innovations have been adopted by Network centers and affiliated providers, and what factors are associated with adoption?
What is the level of collaboration among Network members, and how does collaboration influence center development and outcomes?
What impact has the NCTSI had on the knowledge and delivery of trauma-informed services beyond the NCTSN?
What evidence-based practices have been developed and are currently being disseminated by the NCTSN through registration in the National Registry of Evidence-based Programs and Practices?

The cross-site evaluation’s design includes participation in one or more of eight study components that employ both qualitative and quantitative methods to comprehensively examine the impact of NCTSI funding. This evaluation provides the opportunity to advance the understanding of clinical outcomes among children served in the NCTSN, systematically assess the development and dissemination of evidence-based treatments, and examine in greater detail specific efforts and goals of the NCTSI. The eight study components are as follows.

Descriptive and Clinical Outcomes. To address the GPRA goals of increasing access to services and improving outcomes, the cross-site evaluation utilizes descriptive and clinical outcome data to describe the characteristics of children in formal treatment at NCTSN-funded centers, monitor the type and amount of services that they receive, and assess whether children’s outcomes improve over time. The focus is on children and families receiving intensive treatment for trauma exposure. The cross-site evaluation approach is built on the Core Data Set, collected by the NCCTS by providing guidance, support, and monitoring of that data collection activity. The Core Data Set includes instruments specifically designed for this initiative as well as standardized checklists from the field. The characteristics of clients are summarized using descriptive statistics; understanding the different characteristics and needs of clients is accomplished through multivariate analyses, such as cluster analysis and latent class analysis; client change through treatment is assessed with Reliable Change Index scores; and individual and center effects are measured with hierarchical linear modeling. Data are being collected from children and caregivers at entry into services and at 3-month intervals, for the duration of 1 year.

Satisfaction Study. The cross-site evaluation is assessing the Network’s goal of increasing access and capacity of trauma services to children and their families with an examination of service satisfaction among clients receiving direct clinical mental health services. This survey is administered to caregivers of children who have received direct mental health services from a Network center at the close of treatment or at 6 months into treatment, whichever occurs first. This survey is administered to each caregiver one time, using mixed methods of telephone interviews and hard-copy mailout. All caregivers who have consented to participate in the Descriptive and Clinical Outcomes Study are invited to participate. These data are analyzed with descriptive statistics and compared by demographics of the family consumers, funding cohort, target population and maturity level of the center, and historical and clinical characteristics of the client.

Knowledge and Use of Trauma-informed Services. This study component assesses the extent to which funded Network centers enhance the trauma-informed service knowledge base and use among service providers affiliated with the Network through training and outreach activities. Centers participate in the Trauma-informed Services (TIS) Survey by administering it to human service providers after training and outreach events. The data are analyzed with descriptive and inferential statistics compared by provider demographics, funding cohort, and center target population. In addition, change in knowledge base and use of trauma-informed services will be analyzed longitudinally, as more data are collected.

Product/Innovation Development and Dissemination. This component of the cross-site evaluation identifies and describes the products developed and disseminated to Network and non-Network partners. Three methods of data collection are used in this study component: the Product/Innovation Development and Dissemination Survey (PDDS), telephone interviews with existing NCTSN collaborative workgroup/taskforce coordinators (chairpersons), and case studies. The PDDS is included and completed as part of centers’ quarterly progress reports and the combined fourth quarter/annual report. These reports are completed by project directors or staff from both TSAs and CTSs. Coordinators (chairpersons) of active collaborative workgroups participate in telephone interviews conducted every other year. Five case studies focusing on the development and dissemination of specific Network products/innovations are conducted biannually (in years alternating with the collaborative workgroup coordinator interviews). Five telephone or in-person interviews are conducted for each product selected as a case study. Respondents include key informants who are knowledgeable about the development and dissemination of each of these products. The collaborative workgroup coordinator interviews and the case studies occur in alternating years.

The PDDS is incorporated into the Network’s current quarterly and annual reporting process and provides an updated list of all Network products and innovations developed each year. The survey provides general descriptive information regarding the development and dissemination process. The workgroup coordinator telephone interviews examine the role and impact of the Network’s collaborative workgroups in the development and dissemination of products and innovations. Information obtained from the PDDS and the workgroup coordinator telephone interviews is used to inform case studies conducted in the subsequent year of the evaluation. These case studies provide descriptions of the development and dissemination of Network products and identify best practices in this area. The case studies include telephone and face-to-face interviews with individuals identified as playing key roles in the development and dissemination of Network products.

Both descriptive statistics and thematic qualitative analysis are used to analyze all data collected on product/innovation development and dissemination. These data may be analyzed longitudinally, as well as included in hierarchical models to test for variation by center type and descriptive and outcome data. Qualitative analysis of case studies focuses on identifying best practices for the development and dissemination of products/innovations.

Adoption of Methods and Practice. This component of the cross-site evaluation is designed to evaluate the extent to which trauma-related practices, knowledge, methods, and products, particularly products created or disseminated by the NCTSN, are being adopted by Network centers and non-Network partners. The information obtained through this study enhances understanding of the pathways through which adoption and implementation occur, common barriers, and best practices leading to successful adoption and implementation. The study design consists of a two-stage data collection effort: (1) an annual Web-based survey of all NCTSN centers to determine the types of trauma-related products that are in the process of being adopted, as well as the factors affecting the adoption and implementation of them, and (2) subsequent telephone interviews with a subset of centers to collect additional in-depth, qualitative information about the factors that hinder or support the process of adoption and implementation.

Network Collaboration. This component measures the extent and nature of collaboration among centers by examining how collaboration is used as a conduit for sharing and transferring knowledge, resources, and technology to achieve NCTSI goals. Data are collected from key personnel at each funded center in alternating years of the evaluation via a Web-based survey about the extent to which they interact with every other center on select key activities, such as governance and decision making; information and resource sharing; coordinating activities; product/innovation development; professional training; consumer and client training; and increasing public awareness. Another Web-based survey is administered in the off years to quantify the activities and impact of formal collaboration structures in the Network.

National Impact. This component of the cross-site evaluation examines the extent to which the existence of the NCTSN has impacted trauma-informed services information, knowledge, policy, and practices among mental health and non–mental health child-serving agencies external to the Network. The Web-based National Impact Survey collects data about these agencies’ knowledge and awareness of childhood trauma and practices, about their knowledge and connections to the NCTSN centers, and about their policies, practices, and programs targeted to children and adolescents who have been exposed to traumatic experiences. Findings from this component assist in assessing the role of the NCTSN in diffusing trauma-informed care beyond NCTSN communities.

National Registry of Evidence-based Programs and Practices (NREPP) Review. The NREPP was created by SAMHSA’s Center for Substance Abuse Prevention as part of an effort to help policymakers, consumers, and providers learn more about science-based prevention programs and as a mechanism for disseminating such programs to the field. For the cross-site evaluation, the progress of grantees in submitting practices for NREPP review, as well technical assistance provided by the centers, is tracked. In addition, grantees are monitored in the field and through NREPP to confirm that evidence-based programs developed by or through the NCTSN have been submitted to NREPP for review and possible inclusion in the registry, or are working toward such a submission. The information about NREPP submission and inclusion is organized and warehoused. Although clearance is not requested for data associated with this study component (as they constitute no additional burden for staff or families), the study is mentioned here in order to describe the full scope of the cross-site evaluation.

Clearance Request

SAMHSA is requesting approval for revisions to the previously approved cross-site evaluation package. Changes requested are described below.

The original OMB clearance was requested and approved for the first 3 years of the evaluation. Respondent burden for the revised clearance is calculated for the next 3 years of data collection.
The number of centers for which burden was calculated remains at 44. However, this is simply an estimate, as the numbers of centers active per year changes as new grants are awarded. For the first year of this approval, there will be 51 active centers. After the first year, in September 2009, the 27 grantees funded in 2005 will reach the end of their data collection. At that point, additional centers may be funded or funded again. Because of this variability, the estimate of 44 centers is used, as included in the original OMB package.
Modifications were made to the TIS Survey after it was approved as an amendment to the original OMB approval. In response to feedback from centers that had administered the survey, the survey was shortened significantly by removing several questions, resulting in a reduction in estimated burden. The revised instrument is presented in Attachment 3.G.
Changes in the Product Development and Dissemination component include a change in the schedule and method of completing the PDDS. The survey was originally contemplated as being administered annually as a stand-alone instrument. It was modified to be included as part of the quarterly progress reports and the combined fourth quarter/annual report completed by centers. The numbers relating to respondent burden have been changed accordingly. Also, as a result of the restructuring of the collaborative workgroups in fiscal year 2005, the number of active workgroups is less than 35—the number that was originally anticipated. Estimates in this package are based on the expectation of 15 active workgroups.
Changes in the Adoption of Methods and Practices Study involve a new method of recruitment for service providers. Originally, providers were recruited through the distribution of postcards following center-sponsored training and outreach events. Providers would then self-identify online following the postcard instructions. Now, TIS Survey respondents can indicate a willingness to participate in additional cross-site evaluation surveys. This increased pool of identified service providers will serve as potential respondents for the adoption study survey.

A2. Purpose and Use of Information

This evaluation serves several purposes: (1) collect and analyze descriptive, outcome, and service experience information about the children and families served by the NCTSN; (2) assess NCTSN research, training, and dissemination activities and their impact; (3) assess Network collaboration; and (4) assess the Network’s broader impact. Exhibit 1 illustrates the relationship between the evaluation goals, what is evaluated to assess each goal, and the purpose and utility of evaluation findings. The exhibit also highlights the enhanced evaluation capacity that results from the cross-site evaluation. By providing centers with data, assistance on how data can be used to meet local and Network needs, and, ideally, increasing the appreciation for evaluation and ability to conduct evaluation activities, evaluation findings can be more effectively utilized and evaluation activities can be sustained. The collected data also are useful to SAMHSA, other Federal agencies, individual children and their families, and the research field.

EXHIBIT 1

N
ational Child Traumatic Stress Initiative Evaluation Goals and Utility

SAMHSA can use the results from the evaluation to develop policies and provide guidance regarding the development of the NCTSN. Information and findings from the evaluation can help SAMHSA plan and implement other efforts related to trauma services. SAMHSA also can use the findings from the evaluation to provide objective measures of its progress toward meeting targets of key performance indicators put forward in its annual performance plans as required by law under GPRA. The GPRA indicators that are required to be tracked by the cross-site evaluation are the number of children and adolescents reached by improved services (i.e., whether this number is increasing) and children’s outcomes (i.e., whether outcomes are improving). The measures used in the Descriptive and Clinical Outcomes Study of the evaluation address the GPRA indicators. Specifically, these measures include the Child Behavior Checklist 1.5-5/6-18 (CBCL 1.5-5/6-18); various Core Clinical Characteristics Forms (Baseline Assessment Form, Follow-up Assessment Form, General Trauma Information Form, and Trauma Detail Form), the Trauma Symptom Checklist for Children—Abbreviated (TSCC-A), and the UCLA-PTSD Index (UCLA-PTSD).

Findings from the evaluation can be used by grantees to improve the services, processes, and functions of their centers. Demographic and outcome data on a sample of children and families who participate in the Network aid grantees in identifying the program elements that help children and families function better and that lead to client satisfaction. Grantees can use the information gathered to better identify their target populations and improve their services. Evidence-based treatment development, dissemination, and adoption data can help grantees understand the program processes being implemented, factors that facilitate and hinder these processes, and approaches that can be used to modify existing approaches. Data on Network collaboration can be used to strengthen network relationships by identifying cooperating entities. National impact data can be used to assess the impact of the Network on the broader communities and to determine if goals related to increased access to, and provision of, evidence-based trauma services are being met.

The research community, particularly the field of children’s mental health services research, will continue to profit in a number of ways from the information gathered. First, evaluation of the NCTSI adds significantly to the developing research base about the use of trauma-informed services. Second, the focus on child and family outcomes allows researchers to examine and understand who is being treated for trauma-related problems and the outcomes of that treatment. Third, assessment of the process by which evidence-based trauma services and processes are developed, disseminated, and adopted contributes to understanding the barriers and facilitators that affect this process. Finally, the analysis of evaluation data aids researchers in formulating new questions about the Network and helps both service providers and researchers improve the delivery of children’s trauma services.

If these data collection activities are not continued, policymakers and program planners at the Federal and local levels will not have the necessary information to determine whether centers within the network are working collaboratively and whether they are meeting their objective of developing, disseminating, and implementing evidence-based treatments and processes. As well, they will not have detailed knowledge of who is receiving trauma services and the outcomes of these services.

A3. Use of Information Technology

Web-based surveys are used for data collection in the Adoption of Methods and Practices, Network Collaboration, and the National Impact studies. (In the original approval package, the TIS Survey was proposed as a Web-enabled survey, but this methodology was changed in an amendment in 2007). The use of Web-based surveys decreases respondent burden, as compared to that required for alternative methods, such as a paper format, by allowing for direct transmission of the instrument. In addition, the data entry and quality control mechanisms built into the Web-based survey reduce errors that might otherwise require follow-up, thus reducing burden, as compared to that required for a hard-copy administration. As well, respondents can complete the survey at a time and location that is convenient for them. Respondents are recruited through an e-mail invitation that includes an embedded link to the survey Web site’s URL, which further increases the ease of responding.

The following surveys/processes are Web based or have a Web-based option:

General Adoption Assessment Survey (GAAS)
Network Survey
Child Trauma Partnership Tool (CTPT)
National Impact Survey

All of the Web surveys associated with the cross-site evaluation recruit respondents to participate through an e-mail invitation. The e-mail process occurs in four stages: (1) an advance invitation to participate, (2) a formal invitation, which includes the Web site’s URL and unique user name and password, (3) a reminder to all respondents, and (4) a final targeted reminder to nonresponders and those who have only partially completed the survey.

For the Product/Innovation Development and Dissemination Study, the PDDS is incorporated into the NCTSN’s current quarterly and annual progress reports. Reporting includes annual completion of the annual progress report form, which is a Microsoft Word form that is e-mailed to the project director at each center. These completed forms are e-mailed back to the NCCTS Monitoring and Evaluation Team, and the data are shared with the cross-site evaluator. Other data for this study are collected via telephone interviews with collaborative workgroup coordinators and from telephone and in-person interviews with Network and non-Network key informants involved in the development and disseminations of products/innovations selected as case studies. Responses offered during the telephone and in-person interviews are recorded by members of the evaluation team, thereby eliminating the need for respondents to complete or return by mail any questionnaires or surveys. Respondents are, however, be provided with the interview questions in advance to facilitate their participation in this study.

A4. Efforts to Identify Duplication

While still understudied, research on the effects of trauma on children has been slowly building over the 20th century and into the current century. To identify data on children’s trauma, the cross-site evaluation team conducted a thorough literature review. As well, the evaluator conducted meetings with experts in the field, including researchers from the NCCTS, and attended a national meeting on trauma. Results of these efforts indicated that while data have been collected on children’s trauma, aspects of the field have not been researched or have failed to have large-scale, systematic data collection. This evaluation includes data collection on those areas that have not been studied sufficiently, if at all.

Research that has been conducted includes case studies of children’s reactions to traumatic events, such as surgeries or natural and industrial disasters. These data provided descriptive information about child symptoms that predated, and later correlated to, the diagnostic criteria for PTSD in adults, first outlined in the Diagnostic and Statistical Manual of Mental Disorders (3rd edition) (DSM-III) (American Psychiatric Association, 1980). In the mid-1940s, David Levy (1945), in observing children’s recovery from operations, was perhaps one of the first to suggest that children exhibit traumatic responses similar to those of adults. In 1976, Lenore Terr conducted the first major child trauma research project that was controlled and prospective. Terr (1991) described four major characteristics that she found to be specific to children’s experience of trauma.

Although child trauma research has steadily advanced since Levy’s observations in the 1940s, significant gaps remain, including an understanding of the variability in individual responses to traumatic stressors. Many researchers have recorded differences in children’s responses to traumatic events. The National Comorbidity Survey found that none of the severe traumatic events reported invariably produced PTSD in those exposed to it, and particular types of traumatic events did not necessarily affect different sectors of the population in the same way (Kessler et al., 1995). Other studies also have noted this variability and identified risk factors that affect the likelihood that children exposed to trauma will develop trauma-related symptoms (Breslau, 2001; Chemtob et al., 1997; Davis, 2000; Earls, Smith, Reich, & Jung, 1988; Lloyd & Turner, 2003; Pfefferbaum, 1997; Pynoos et al., 1987; Terr, 1991).

Although many risk factors have been identified, much work remains to improve the understanding of protective factors and treatments that address the range of children’s individual experiences. A 2000 literature review of PTSD in children observes that “only recently has the mental health community recognized the applicability of diagnostic criteria for PTSD in children and adolescents, including a consideration of age-related features” (Davis, 2000, p. 135).

This evaluation generates data that have not previously been collected, or have only minimally been collected, in the field of children’s trauma. This includes information on the development, dissemination, and use of evidence-based treatments (EBTs) in trauma services; the impact of Network-developed/promoted EBTs on trauma services within and outside the NCTSN; and the processes by which distinct centers that provide and support trauma services collaborate. As well, data on who receives trauma services, the types of services they receive, and the outcomes related to receipt of these services are collected in a systematic manner that yields more extensive, detailed, and consistent information than has previously been obtained.

In sum, existing research and data in the area of child trauma are not sufficient to address the questions posed in this evaluation. For questions related specifically to the functioning and impact of the NCTSN, no data exist. For questions related to descriptive and clinical information on children receiving trauma services, existing data have not been consistently collected such that the data are representative and comparable across service environments. While data have been collected on EBTs in general, data do not exist on the development and use of EBTs in treating child trauma, nor specifically on the role of the NCTSN in this area. Thus, this evaluation generates new data and will not be reproducing existing data.

A5. Involvement of Small Entities

Most data for this evaluation are collected from service providers, administrators, and researchers affiliated with NCTSN centers, which are public or private agencies that receive funding from the Federal Government and for whom participation in the evaluation is considered to fall within their job responsibilities. Some data are collected from mental health and non–mental health service providers working outside of the NCTSN centers. While most of these data are collected from public agencies, some organizations and individuals providing services to the target population, such as community-based organizations, not-for-profit agencies, or private providers, may qualify as small entities, but not a significant impact.

A6. Consequences if Information Is Collected Less Frequently

Descriptive and Clinical Outcomes Study. Descriptive data are collected when children and families first enter services. As part of their normal operations, grantees collect these data for multiple purposes, including third-party reimbursement verification and for aggregate reporting to various State and local funders. If these data are not collected at entry into services, it would be impossible to identify the characteristics of individual cases and to subsequently analyze the impact that these characteristics have on clinical outcomes.

Outcomes data are collected at entry into services and every 3 months for up to 1 year for a subset of children. Three-month intervals were selected in order to capture changes after initial entry into treatment and to monitor those changes closely after children transition out of intensive treatment or out of treatment altogether. Although many children will experience significant improvement in the first 3 to 6 months of trauma-focused treatment, it is necessary to continue collecting data at 9 and 12 months in order to understand the maintenance of changes across time. Longer and less frequent data collection intervals would miss important changes that are likely to happen with children during their treatment episode or shortly thereafter. It is necessary to have multiple data collection points to effectively monitor these changes in clinical outcomes.

Consumer Satisfaction. Data for the Satisfaction Study are collected one time per client at the end of treatment or at 6 months into treatment (whichever occurs first) from caregivers of children enrolled in the outcome study (i.e., children enrolled into clinical services). Satisfaction data are critical for quality monitoring at the local and national levels, and less frequent collection of this information would not provide the opportunity for timely, data-driven programmatic improvement by the grantees or SAMHSA.

Adoption of Methods and Practice. Data for the GAAS and the Adoption and Implementation Factors Interview (AIFI) are collected annually. The process of local and national grantees adopting products developed by the Network will extend throughout the life of the program. New products will be developed and existing products will be improved during the life of the program. The sites adopting the products will be exposed to new products and will elect to adopt products as they are developed and refined during the life of the program. Multiple data collection points are needed in order to assess the degree of proliferation of Network products and the extent to which products are being utilized over time.

Network Collaboration. Network collaboration data are collected annually for the duration of the evaluation, with each of the collaboration instruments administered in alternate years. The Network Survey was administered in the first and third years and will continue in odd alternating years of the evaluation, while the CTPT was administered during the second and fourth years and will continue in even years. The Network Survey utilizes social network analysis techniques (Wasserman & Faust, 1995) to inquire about the extent to which each NCTSN center interacts with every other center on select key NCTSN activities (governance/decision making, information sharing, coordination of activities, product development, product dissemination and adoption, and training and technical assistance). The CTPT was designed to assess the activities and impact of the NCTSN collaboration structures (workgroups, committees, consortia) in terms of membership activities, accomplishments, formalization, leadership, communication, vision, decision making, resource allocation, and understanding/valuing. It is expected that there will be greater collaboration over time as relationships among NCTSN centers become more interdependent and as the formal workgroups mature and become productive. Therefore, measuring relationships among centers (Network Survey) and the organization and performance of workgroups (CTPT) in alternating years will provide the minimum frequency of data collection required to assess change over time and whether goals of the NCTSI, regarding the vital role of collaboration in information and technology transfer, are being met.

Provider Knowledge and Use of Trauma-informed Services. TIS Survey data are collected from providers affiliated through training activities with the NCTSN after center-sponsored training and outreach events. Increased awareness and use of trauma-specific services among child service providers is critical to the overarching mission of the NCTSI to increase the quality and access of care for children who experience trauma. Less frequent data collection would result in the inability to understand the extent to which the Network and its centers are enhancing the understanding and increasing the use of appropriate services for children who experience trauma. It may also result in a sample of training events that isn’t representative of all NCTSN training activities, leading to potentially biased results. Ongoing data collection is needed to assess the change in the knowledge base and use of trauma-informed services as the Network and its affiliated centers mature.

Product Development and Dissemination. The PDDS has been incorporated into centers’ quarterly progress reports and the NCTSN’s current annual progress reporting system. Interviews with collaborative workgroup coordinators were conducted in years 1 and 3 of the evaluation and will continue in alternating years. Five case studies that will include telephone and face-to-face interviews with individuals involved in the development and dissemination of Network products and innovations will continue to be conducted in alternating years opposite the workgroup coordinator interviews. The schedule described above minimizes respondent burden by limiting the frequency of data collection to only what is necessary to adequately describe and assess the product development and dissemination process within the NCTSN. Multiple data collection points are needed to maintain a current inventory of Network products and to examine how Network strategies and approaches develop over time.

National Impact. To evaluate national impact, the Web-based National Impact Survey is administered annually to assess change in agencies’ policies and practices towards more trauma-informed care. The survey is administered to children’s mental health agencies and non–mental health child serving agencies (e.g., education, child welfare, justice) in alternating years. This data collection schedule provides the minimum frequency of data collection required to assess impact beyond the NCTSN, and minimizes burden for respondents. Less frequent data collection would limit our ability to assess the impact, over time, of the NCTSN on trauma-informed care beyond the NCTSN.

A7. Consistency With Guidelines of 5 CFR 1320.5

The data collection fully complies with the requirements of 5 CFR 1320.5(d)(2).

A8. Consultation Outside the Agency

SAMHSA published a notice in the Federal Register on Monday, January 26, 2009 (pg 4442) soliciting public comment on this study. SAMHSA received no comments on the planned data collection.

Consultation on the design, instrumentation, data availability and products, and statistical aspects of the evaluation occurred throughout the development of the evaluation design process and throughout the first 3 years of the evaluation. Consultations have been sought from the following:

The Federal Government
Experts in collaboration
Experts in development, dissemination, and adoption
Experts in logic modeling
Experts in cultural competence
Family representatives
Family members (i.e., families receiving services in the NCTSN)
Network staff

These consultations had several purposes: (1) to ensure coordination and collaboration of the cross-site evaluation with the NCCTS; (2) to ensure the rigor of the evaluation design, the proper implementation of the design, and the feasibility of implementation; and (3) to verify the general relevance of the data to be collected and their specific relevance to families and members of minority groups.

Some of these consultants worked with the cross-site evaluator over the course of several months to design the evaluation and its study components (i.e., the experts in collaboration; logic modeling; and product development, dissemination, and adoption). (Attachment 1.B provides a list of expert consultants.) In addition, Network staff provided feedback and input into the evaluation design during eight site visits to a selection of NCTSN centers. (Attachment 1.E provides a list of sites visited.)

Consultation was also sought upon completion of the draft evaluation design via presentation to various consultant groups and solicitation of structured feedback. For example, the cultural competence consultants who constitute the Cultural Competence Review Committee met with the evaluator on March 17, 2005, to review the draft evaluation design and provide feedback. (Attachment 1.C provides a list of Cultural Competence Review Committee members.) The family representatives and family members attended a Family Review Committee meeting in Atlanta, GA, on March 15 and 16, 2005, at which time they were presented the evaluation design and solicited for targeted feedback. (Attachment 1.D provides a list of Family Review Committee members.) In addition, NCTSN representatives (i.e., NCCTS staff, Network Steering Committee members) provided feedback on the evaluation design and its proposed instruments.

All instruments and guides, with the exception of the TIS Survey and the AIFI, which were developed later, underwent cognitive testing, pilot testing, expert review, or usability testing during July and August of 2005. The feedback from these efforts was incorporated into this supporting statement and the proposed instrumentation of the cross-site evaluation. The TIS Survey and the AIFI were both developed during 2007 and underwent expert review and pilot testing at that time.

Finally, SAMHSA, and in particular the Project Officer for the cross-site evaluation, provided ongoing input and review of all aspects of the evaluation design. The Federal Project Officers assigned to each of the funded centers reviewed the draft evaluation design and provided comprehensive written feedback on each component. (Attachment 1.A provides a list of Federal consultants who reviewed the initial evaluation design.)

All feedback was requested in a systematic way and provided in a structured and written format. The feedback received was considered and included, as appropriate, in the revision of the evaluation design and its associated instruments.

A9. Payment to Respondents

Remuneration is not provided by the cross-site evaluator to respondents for the majority of study components. Many of the respondents who will be providing data work in an NCTSN center and receive wages from the NCTSI grant, which is federally funded. These respondents are not eligible to receive additional remuneration for participating in the evaluation. Study components that do involve remuneration are the trauma-informed services study, the product development and dissemination case studies, the Adoption of Methods and Practices Study, and the Satisfaction Study.

For the satisfaction with direct mental health services survey, participants are being asked to self-identify and then participate in the satisfaction survey. In order to promote self-identification and encourage completion of the survey, the cross-site evaluation remunerates caregivers for their time with a $20 money order. This amount of remuneration is suggested to achieve response rates of 80% and to appropriately compensate caregivers for their time and contribution.

For the TIS Survey, training participants are thanked for taking the time to complete the survey and are reminded that their participation is critical to expanding the knowledge base related to trauma-informed service provision. In addition, respondents are informed that, having completed the survey, they are now eligible to participate in a lottery drawing for a $50 gift certificate from Amazon.com. If they would like to be entered into the lottery, they must complete the contact information form on the last page of the survey. One $50 gift certificate will be provided to a participating respondent of each training event. The opportunity to participate in the lottery serves as remuneration for completing the TIS Survey. Each individual who submits a contact information form will be entered into the lottery for each training event, one winner will be randomly drawn, and the $50 gift certificate will be mailed or e-mailed to the respondent. The potential respondents for the TIS Survey are affiliated with Network centers through training activities, but not necessarily receiving wages through the NCTSI grants.

For the product development and dissemination case studies, caregivers participating in the case study interviews are paid $25 for their time. Payment is provided once the participant has agreed and provided consent. Participants can withdraw from the interview at any time and still retain payment.

For the AIFI, staff members who are employed by formerly funded centers (i.e., centers that are no longer funded by the SAMHSA grant) are paid $40 for their time. Participants provide consent just before the start of the interview; following the interview, they are provided with the payment. Participants can withdraw from the interview at any time, refuse to answer some questions, or refuse to be recorded, and still retain payment.

Payments to respondents for participating in descriptive and outcomes data collection is at the discretion of each individual center participating in the NCTSN. Prior to the evaluation, centers collected this information for clinical evaluation purposes only and have not paid respondents. For evaluation data collection, the cross-site evaluator has strongly recommended that centers pay respondents for participation in each data collection interval. The cross-site evaluation team recommends that centers pay families $10 for participation in each data collection interval (approximately 45 minutes), resulting in a total of $50 for participation in all five data collection points.

Remuneration is standard practice in this type of longitudinal research to acknowledge participants’ value to the study. It is essential to help maximize participation rates, particularly given the additional time being asked of families who already face multiple challenges and demands on their time in caring for their children. Caregivers and children who participate in the Descriptive and Clinical Outcomes portion of the evaluation are asked to complete more assessments than are ordinarily required in the course of receiving services. To complete the instruments at the time of entry to services and at subsequent follow-up points requires the evaluation participants to spend time away from other activities. The combination of the number of instruments and their periodicity creates a burden to the caregivers and children that exceeds the burden that ordinarily would be placed on them if they were seeking services not associated with this evaluation. This approach of recommending to sites in a cross-site evaluation that they pay respondents from whom the sites collect data directly was used in other evaluations cleared by OMB (e.g., OMB numbers 0930-0171, 0930-0192, 0930-0209, and 0930-0257).

A10. Assurance of Confidentiality

For all studies in the cross-site evaluation, all reports and publications from these data include only group-level analyses that fully protect the privacy of individual participants, and no data have been or will be stored with identifying respondent information.

Descriptive and Clinical Outcomes. This portion of the evaluation requires collecting descriptive and clinical data from children and families. In all grantee sites, data are collected by site staff. These staff members are responsible for developing procedures to protect the privacy of all participants in the evaluation data collection, storage of data, and reporting of all information obtained through data collection activities. These procedures include limiting the number of individuals who have access to identifying information, using locked files to store hard-copy forms (if used), assigning unique code numbers to each participant to ensure anonymity, and implementing guidelines pertaining to data reporting and dissemination.

Data from caregivers and youth are collected through interviews by site staff. The content of some questions is sensitive in nature, and some participants may experience psychological or social distress during an interview. The cross-site evaluation team provides guidance to local staff through procedures manuals and training to assist communities in establishing local interviewer training to address respondent distress and other circumstances that may arise during an interview. Local evaluators develop procedures appropriate to local requirements, including guidelines for referral to requested services, and report abuse, neglect, and harm to self or others according to local law.

Each grantee implements an active consent procedure that informs the participants of the purpose of the evaluation, describes what their participation entails, and addresses the maintenance of privacy as described above. Informed assent is obtained from participating older children and adolescents (ages 7–17). In addition, informed consent is obtained from adolescents who have reached the age of 18 at follow-up data collection. Written informed consent/assent is obtained from children and families at the point of entry into services. Given that some children targeted for study recruitment may be the victims of ongoing domestic violence and their treatment status may be unknown to the perpetrator, special considerations will be taken around the signing of physical consent forms (e.g., when the signing of a consent form leaves a “paper trail” that may put the child or other study recruit in harm’s way, verbal consent procedures can sometimes be approved and secured through local institutional review boards [IRBs]) and methods for contacting the family for follow-up data collection interviews (e.g., using alternative methods of contact or disguised interviewer identity). Each grantee obtains local IRB approval for the informed consent procedures used in this evaluation. To further protect evaluation participants, all grantees are asked to obtain a Federal Certificate of Confidentiality, authorized by Section 301(d) of the Public Health Service Act in order to provide additional protection of the information about the participants from civil and criminal subpoena.

To further protect study participants, the cross-site evaluator obtained a Federal Certificate of Confidentiality, authorized by Section 301(d) of the Public Health Service Act. This certificate provides additional protections of the data from civil and criminal subpoena. Additionally, the cross-site evaluator conforms to all requirements of the Privacy Act of 1974, under the System of Records: Alcohol, Drug and Mental Health Epidemiological, and Biometric Research Data, U.S. Department of Health and Human Services (HHS), #09-30-0036; the most recent publication in the Federal Register occurred on January 19, 1999 (64 FR 2914). Client records at the sites are also covered under this Privacy Act System of Records. In addition, the cross-site evaluator obtained a Federalwide Assurance (FWA), which ensures compliance with U.S. Federal regulations for protection of human subjects in research including the Common Rule (Title 45 Code of Federal Regulations Part 46) and other regulations as applicable. The cross-site evaluator also requests that all grantees obtain an FWA.

Consumer Satisfaction. In order to maintain anonymity, caregivers self-identify and provide contact information to the evaluation contractor survey center staff who oversee this data collection effort. The survey center maintains separate databases for the contact information and the survey data. The survey center sends the cross-site evaluator the data linked to the respondent ID but with no personal identifiers. Every month, Network centers send the evaluator the survey respondent ID from the invitation linked to the ID from the Descriptive and Clinical Outcomes Study, but they do not send any personal identifiers. It is necessary to include the respondents’ Descriptive and Clinical Outcomes Study ID so that the evaluator can analyze the satisfaction survey data in conjunction with the outcomes data. For example, the outcomes study contains information on demographics and how long the child and family has been receiving services, and this can be analyzed to determine if there are associations between satisfaction with services and length of time in services, race, gender, age, and income. After the data have been sent to the cross-site evaluator, the survey center destroys the database that contains the contact information. The survey center administers a verbal consent when the respondents are contacted, and the survey is conducted only if the respondent consents to participate.

Participants are asked their preferred survey mode (mail survey or telephone) and are administered the survey according to their preference. There are separate databases with contact information and survey data, which are linked with a unique, randomly generated ID. This random ID will be used for tracking purposes only, and the link is destroyed as soon as the survey is returned to the evaluator. However, the study ID cannot be linked with any personal identifiers because the link to contact information is destroyed when the surveys are returned and the study ID is contained in the contact database. As well, when surveys are received by the evaluator, the survey is immediately separated from the envelope (which may have the respondents’ return address).

Adoption of Methods and Practice. Respondents to this survey include direct service providers employed by Network centers, service providers affiliated with the Network through training and outreach activities centers’ program director or administrator, and the program evaluator. Consent is obtained from respondents before they complete the Web-based GAAS or participate in the AIFIs. In the case of each data collection effort, the initial recruitment letter explains the survey, including the voluntary nature of survey completion, anonymity of responses, and the risks, benefits, and rights as respondents, and it advises the recipient that by completing and submitting the survey or participating in the interview, they are indicating consent. Information about the study and participant rights also is presented at the start of the GAAS and is sent to each interview participant before the interview.

The security and anonymity of data entered and managed on the Web-based GAAS are assured. Access to the GAAS is password protected, and the GAAS uses data encryption to further enhance security and protect privacy. To maintain anonymity of responses, two databases have been created for the GAAS: one to store the identifying information, including name, user ID, and password, and the other database to store the survey responses. The two databases are not linked after the data are collected. While data are being collected, only the system administrator has the key that links the two databases, and this key is destroyed when the data are transmitted to the evaluator.

For the GAAS, respondents are asked to log in using an assigned ID and randomly generated password that is provided in the formal invitation. After the respondent logs on to the survey, it is possible to check off that the subject responded to the survey in the identifier database.

For the AIFI, in advance of each interview, respondents are provided a confirmation of the date and time of the interview, which includes background information about the purpose of the study component, the purpose of the AIFI telephone interview, the risks and benefits of participating, privacy and consent information, and respondent selection criteria. A consent form is attached to the document describing these points (Attachment 2.C.4). Before conducting the survey, the interviewer reviews the informed consent form with the respondent, verifies that the respondent agrees to consent to the interview, and advises that by completing the interview, the respondent is indicating consent.

Network Collaboration. The Network Survey and the CTPT are administered to individuals directly affiliated with the NCTSN through a Web-based format. Respondents of the Network Survey are NCTSN center directors and center associate directors. Respondents for the CTPT are members of the approximately 15 workgroups (the number of workgroups could increase to about 25 depending on how many new groups are added each year) that have been assembled by the NCTSN. Full contact information for respondents, including name, address, phone, and e-mail addresses, is gathered from the SAMHSA Program Officer and from the NCCTS, which tracks workgroup activities for the Network. Respondents are recruited to participate through an e-mail invitation.

The surveys are administered in alternating years throughout the evaluation, but respondents may be different at each point because of possible changes in center directors or associate directors over time (Network Survey) or changes in the membership of workgroups over time (CTPT). Passive consent is obtained. The formal invitation explains the survey, including the voluntary nature of survey completion, privacy and anonymity of responses, and the risks, benefits, and rights as respondents, and advises the recipient that completion and submission of the survey indicate consent to participate. This invitation also provides contact information if the survey recipient has questions or desires clarification prior to participation.

Access to the Network Survey and CTPT is password protected, and both surveys use data encryption to further enhance security and protect privacy. To maintain anonymity of responses, two databases have been created for each survey: one to store the identifying information, including name, user ID, and password, and the other database to store the survey responses. The two databases are not linked after the data are collected. While data are being collected, only the system administrator has the key that links the two databases, and this key is destroyed when the data are transmitted to the evaluator.

Respondents are asked to log in using an assigned ID and password that is provided in the formal invitation. After the respondent logs on to the survey, it is possible to check off that the subject responded to the survey in the identifier database.

Provider Knowledge and Use of Trauma-informed Services. The TIS Survey is administered to providers affiliated, through training activities, with NCTSN centers. The paper survey is distributed to all participants of center-sponsored training and outreach events. A consent form is included as the first page of the TIS Survey and explains the survey, including the voluntary nature of survey completion, privacy of responses, and the risks, benefits, and rights as respondents, and it advises the recipient that completion and submission of the survey indicate consent to participate. In addition, as described previously, TIS Survey respondents are invited to provide their contact information if interested in participating in the TIS Survey respondent lottery or in authorizing the use of their contact information for future cross-site evaluation surveys regarding the implementation of trauma-informed services. Contact information provided by respondents is collected separately from completed TIS Surveys by the NCTSN center trainer to ensure anonymity. A tracking code is included on each TIS Survey that matches the code printed on the attached contact information form (i.e., for each individual TIS Survey and attached contact information form, tracking codes printed on each will match). The code is used for two purposes: (1) to ensure that the lottery winner submitted a TIS Survey and (2) to link TIS Survey data with data collected through other cross-site evaluation surveys designed to assess the implementation of trauma-informed services. Linking respondent participation across surveys provides an opportunity to examine the relationship between respondents’ exposure to NCTSN training and outreach events and the implementation of or change in trauma-informed services provision over time. The contact information and TIS Survey data are stored in separate databases and will never be linked.

Product Development and Dissemination. With the exception of the PDDS, which is completed as part of the Network’s regular progress reporting process, informed consent is obtained for each instrument administered for this component of the evaluation and for the audiotaping of any interviews. Identifying information is only used to contact respondents and schedule interviews. Any information used to identify respondents is maintained in a contact database that remains separate from the database that stores data from each completed interview. For each interview, the evaluation team assigns the respondent a nominal ID.

National Impact. The Web-based survey is administered every 12 months as a cross-sectional assessment of agencies’ policies and practices. In alternating years, the survey is administered either to children’s mental health agencies (years 1, 3, 5, etc.) or to child welfare, justice, and education agencies (years 2, 4, 6, etc.). Active consent is obtained at each wave of survey administration from respondents who are the agencies’ executive directors or directors of children’s services, depending on who is most knowledgeable about their agencies’ polices/practice and relationships with other agencies in the service system.

Full contact information for respondents, including name, address, phone, and e-mail addresses, is assembled from the membership rosters of professional organizations representing mental health, child welfare, education, and juvenile justice agencies. The cross-site evaluator establishes cooperative agreements with these professional organizations to access their roster information where possible.

Respondents are recruited to participate through an e-mail invitation. The formal invitation explains the survey, including the voluntary nature of survey completion, anonymity of responses, and the risks, benefits, and rights as respondents. This invitation also provides contact information if the survey recipient has questions or desires clarification prior to participation. The second page of the survey contains an informed consent form that asks the potential respondents to certify (by checking a space for “agree” or “do not agree”) that they have read the informed consent form, understand its content, and freely agree to participate in the project.

Access to the National Impact Survey is password protected, and the survey uses data encryption to further enhance security and protect privacy. For anonymity of responses, two databases are created for the survey: one stores the identifying information, including name, user ID, and password, and the other database stores the survey responses. The two databases are not linked after the data are collected. While data are being collected, only the system administrator has the key that links the two databases, and this key is destroyed when the data are transmitted to the evaluator.

A11. Questions of a Sensitive Nature

Because this project concerns services to children who have experienced traumatic events and their families, it is necessary to ask questions that are potentially sensitive as part of the Descriptive and Clinical Outcomes Study. However, only information that is central to the study is being sought. Questions address dimensions such as suicidality and other self-injurious behaviors, criminal activity, developmentally inappropriate sexual behaviors, negative feelings, and experience of specific types of traumatic events, such as physical/sexual/psychological maltreatment, natural disasters, or terrorism. The answers to these questions are used to understand who is being served by the NCTSN, to determine baseline status, and to measure changes in these areas experienced after receiving NCTSN services. The measures that contain the sensitive questions are from the Network’s Core Data Set and have been selected by, and used in, the Network prior to the evaluation. Thus, the cross-site evaluation is not introducing new, sensitive domains of inquiry.

A12. Estimates of Annualized Hour Burden

In accordance with the evaluation design, data collection for the 19 CTSs, 8 TSAs, and 2 NCCTS centers funded in 2005, the 10 CTSs and 5 TSAs centers funded or funded again in 2007, and the 8 CTSs funded in 2008 will span the 3 years covered by this reapproval. Centers funded in 2005 will stop their participation in the evaluation at the end of their grant funding period, in September 2009. Similarly, centers funded in 2007 will not participate in data collection after September 2011. Additional centers also may be funded in future years, and they will be incorporated into the evaluation as they are funded. Because of the variability and uncertainty in the number of funded centers in each year, burden estimates are calculated on the basis of the 44 centers, as was the case in the original OMB approval package. Based on the data collection experience during the first three years of this evaluation, we estimate that only 75% of centers are eligible for participation in the Descriptive and Clinical Outcomes and Satisfaction studies, based on variation in programmatic focus across centers. As a result, burden estimates for those two studies are based on an estimate of 33 participating centers.

Table 1 shows the burden associated with the cross-site evaluation during years 4–6 of the evaluation, the period for which OMB clearance is being sought. Burden estimates presented in Table 1 are based on information supplied by various sources. Measures that are newly developed for this evaluation were piloted by the evaluator to determine average burden estimates and have since been implemented in the field, allowing for updated burden estimates. These measures include the GAAS, National Impact Survey, Network Survey, Product/Innovation Development and Dissemination Interview, PDDS, CTPT, and TIS Survey.

The CBCL 1.5-5 and CBCL 6-18 were used in another OMB-approved study (clearance number 0930-0257), and the estimated average burdens used in that study are used here. The Core Clinical Characteristics Forms, the TSCC-A, and the UCLA-PTSD have been used for data collection by the grantees prior to the cross-site evaluation. Estimated burden times for these measures are based on feedback from the grantees and piloting conducted by the evaluator.

The cross-site evaluation will continue to review contributions to the NREPP by grantees. Because these data will be compiled from the registry and not directly from grantees, they will result in no burden to respondents. Clearance for collection of these data is, therefore, not being requested and is not discussed further in this section.

TABLE 1

Estimate of Respondent Burden

Note: Total burden is annualized over the 3-year clearance period.

Instrument	Number of Respondents	Average Number of Responses per Respondent	Hours per Response	Total Burden Hours	3-year Average of Annual Burden Hours	Hourly Wage Rate ($)		Total Cost per Year ($)
Caregivers
Child Behavior Checklist 1.5-5/6-18 (CBCL 1.5-5/6-18)	2,475¹	5	0.3	4,084	1,361	10.20²	13,882
Trauma Information/Detail Form	2,475	5	0.2	2,723	908	10.20	9,262
Core Clinical Characteristics Form	2,475	5	0.4	4,950	1,650	10.20	16,830
Youth Services Survey for Families (YSS-F)	2,475	1	0.1	198	66	10.20	673
UCLA-PTSD Short Form (UCLA-PTSD)	2,475	5	0.2	2,104	701	10.20	7,150
Case Study Interviews	10³	1	1.5	15	5	10.20	51
Youth
Trauma Symptoms Checklist for Children-Abbreviated (TSCC-A)	1,881⁴	5	0.3	3,104	1,035	6.55⁵	6,779
Service Providers
Provider Trauma-informed Services Survey (TIS)	29,250	1	0.2	5,850	1,950	19.25⁶	37,538
General Adoption Assessment Survey (GAAS) Providers	14,040	1	0.5	7,020	2,340	19.25	45,045
Adoption and Implementation Factors Interview (AIFI) Provider Assessment & Clinical Components	150	1	1.0	150	50	19.25	963
Project Directors/Principal Investigators
Product/Innovations Development and Dissemination Survey (PDDS)	44	12	1.0	528	176	19.25	3,388
General Adoption Assessment Survey (GAAS) Administrators	44	3	0.5	66	22	19.25	424
Adoption and Implementation Factors Interview (AIFI) Administrator Assessment & Clinical Components	45	1	1.0	45	15	19.25	289

TABLE 1

Estimate of Respondent Burden

Note: Total burden is annualized over the 3-year clearance period.

Instrument	Number of Respondents	Average Number of Responses per Respondent	Hours per Response	Total Burden Hours	3-year Average of Annual Burden Hours	Hourly Wage Rate ($)	Total Cost per Year ($)
Network Survey	84	1	1.0	84	28	19.25		539
Other Network Staff
TIS Training Summary Form	1,463⁷	1	.1	122	41	19.25		782
Workgroup/Taskforce Coordinator Interview	15⁸	1	1.5	23	8	19.25		144
Case Study Interviews	20⁹	1	2.0	40	13	19.25		257
General Adoption Assessment Survey (GAAS) Evaluators	44¹⁰	3	0.5	66	22	19.25		424
Adoption and Implementation Factors Interview (AIFI)	30¹¹	1	1.0	30	10	19.25		193
Network Survey	44¹²	1	1.0	44	15	19.25		289
Child Trauma Partnership Tool (CTPT)	200¹³	2	0.8	320	107	19.25		2,053
Non-Network Mental Health Professionals
National Impact Survey	1,600	1	0.5	800	267	19.25		5,133
Non-Network Non–Mental Health Professionals
National Impact Survey	1,600	2	0.5	1,600	533	19.25		10,267
Non-Network Product Developers
Case Study Interviews	20	1	1.5	30	10	19.25		193
Total summary	62,959	61			33,996			487,674
Total annual summary	20,986	20			11,333			162,558

On average, 75 percent of centers are eligible to participate in the Descriptive and Clinical Outcomes Study (33 of 44 centers). At each of these centers, an average of 25 caregivers will participate in each year.
Assuming that most of the families participating in the evaluation sample fall at or below the 2008 HHS National Poverty Level (U.S. Department of Health and Human Services, 2008) of $21,200 (based on a family of four), the wage rate was estimated using the following formula: $21,200 (annual family income)/2,080 (hours worked per year)=$10.20 per hour.
One caregiver will participate in each of the 10 case studies that will be conducted during the clearance period.
On the basis of the children enrolled at centers participating in the cross-site evaluation through June, 30, 2008, approximately 76% of the children in the evaluation will be between the ages of 7 and 18.
Based on the Federal minimum wage rate of $6.55 per hour.
Assuming the average annual income across all types of staff/service providers/administrators is $40,000, the wage rate was estimated using the following formula: $40,000 (annual income)/2,080 (hours worked per year)=$19.25 per hour.
Respondents will be center trainers or evaluation staff. On average, one Training Summary Form is completed for every 20 TIS Surveys (one per training event).
Respondents will be workgroup/taskforce coordinators.
Respondents will be stakeholders.
Respondents will be evaluators.
Respondents will be researchers, supervisors, and administrators.
Respondents will be associate directors.
Respondents will be collaboration structure staff.

As indicated in Table 1, the average total annual burden for data collection is estimated at 11,333 hours. This estimate was derived by calculating the burden for each measure, dividing those numbers by 3 (years of data collection in the cross-site evaluation for which OMB clearance is being requested with this submission), and summing.

A13. Estimates of Annualized Cost Burden to Respondents

There are no startup, capital, and maintenance costs associated with data collection for respondents. Grantees are collecting the data for the Descriptive and Clinical Outcomes Study as part of their normal operations, and they maintain this information for their own service planning, quality improvement, and reporting purposes. The instruments used in this study are those that constitute the NCTSN’s Core Data Set, and these data have been collected by the grantees prior to implementation of the cross-site evaluation.

Each grantee has been funded, as part of the overall cooperative agreement award, to participate in the cross-site evaluation, with up to 20% of the grant award available for evaluation efforts and data collection. Therefore, no cost burden is imposed on the grantee by this information collection effort. Other costs related to this effort, such as the cost of data collection for studies other than the Descriptive and Clinical Outcomes Study, data analyses, and materials, are costs to the Federal Government.

A14. Estimates of Annualized Costs to the Government

SAMHSA has planned and allocated resources for the management, processing, and use of the collected information in a manner that enhances its utility to agencies and the public. Including the Federal contribution to local grantee evaluation efforts, the contract with the cross-site evaluator, and Government staff to oversee the evaluation, the annualized cost to the Government is estimated at $3,577,321. These costs are described below.

Each grantee is expected to participate in the cross-site evaluation, including collection of information for the Descriptive and Clinical Outcomes Study and the Trauma-informed Services Study. Assuming (1) that 44 centers will participate in the Descriptive and Clinical Outcomes Study or the Trauma-informed Services Study, (2) that each of these 44 centers will have 1.5 full-time equivalents (FTEs) dedicated to evaluation, (3) an average annual salary of $30,000 for evaluation staff, and (4) that the average Federal contribution will be 100%, the annual cost for implementing the cross-site evaluation at the grantee level is estimated at $1,980,000. These monies are included in the cooperative agreement awards.

The original cross-site evaluation contract was awarded to Macro International Inc. for evaluation of all NCTSN centers. The cross-site evaluation contract provides for 1 base year of $1,406,740, with an option to renew for 4 more years. The estimated average annual cost of the contract is $1,558,321. Included in these costs are the expenses related to developing and monitoring the cross-site evaluation, including, but not limited to, the following activities: development of the design, instrument package, and training materials; monitoring of and technical assistance to sites; travel to sites and relevant meetings; and data analysis and dissemination activities. Although the original evaluation contract will expire in September 2009, these same cost estimates are applied for the 3 years of the evaluation spanned by this reapproval.

It is estimated that SAMHSA will allocate 60% of an FTE each year for Government oversight of the evaluation. Assuming an annual salary of $65,000, these Government costs will be $39,000 per year.

A15. Changes in Burden

The estimate of annual burden hours associated with the original 3-year approval period was 17,222 (based on the original OMB approval and a subsequent desk review for the TIS Survey and AIFI). The program is requesting 11,333 annual hours for this submission, a reduction of 5,889 annual hours. This revision responds to a variety of minor changes: (1) revision of the TIS Survey to remove 4 pages of content; (2) a change in the method of data collection for the PDDS, and (3) a change in the recruitment of provider respondents for the GAAS. In addition to these changes, the estimates of the number of respondents to the Descriptive and Clinical Outcomes Study instruments, the TIS Survey, and the GAAS were changed significantly based on our data collection experience during the past three years. Also, because several of the evaluation instruments are administered in alternating years of the evaluation, some instruments were administered twice during the original 3-year approval period which will now be administered only once during this reapproval period (Network Survey and workgroup coordinator interviews). Similarly, other instruments were administered once and will now be administered twice during the reapproval period (CTPT, product development case studies). Finally, the TIS interviews and focus groups were conducted during the first 3-year approval period in order to inform the development of the TIS Survey. These will not be conducted again during this approval period. These explain the remaining difference in burden estimates across the two approval periods.

Program Changes

Modifications were made to the TIS Survey after it was approved as an amendment to the original OMB approval. In response to feedback from centers that had administered the survey, the survey was shortened significantly by removing several questions, primarily focusing on the types of trauma covered in the training and the topics covered in the training. These questions were moved to the Training Summary Form. We also reduced by half our estimate of the number of respondents who will be targeted, based on our knowledge of the percentage of trainings that are relevant for TIS Survey distribution. We also added 122 burden hours associated with center completion of the Training Summary Forms. The annual reduction in burden for this study component was 3,859 annual hours.
Changes in the Product Development and Dissemination Study include a change in the schedule and method of completing the PDDS. The survey was originally contemplated as being administered annually as a stand-alone instrument. It was modified to be included as part of the quarterly progress reports and the combined fourth quarter/annual report completed by centers. This resulted in a net increase in burden of 132 hours. Also, as a result of the restructuring of the collaborative workgroups in fiscal year 2005, the number of active workgroups is less than 35—the number that was originally anticipated. Estimates in this package are based on the expectation of 15 active workgroups, yielding a reduction in burden of 82 hours.
Changes in the Adoption of Methods and Practices Study involve a new method of recruitment for service providers. Originally, providers were recruited through the distribution of postcards following center-sponsored training and outreach events. Providers would then self-identify online, following the postcard instructions. Now, TIS Survey respondents can indicate a willingness to participate in additional cross-site evaluation surveys. This increased pool of identified service providers will serve as potential respondents for the adoption study survey. This change results in an estimated increase of 5,370 burden hours.

A16. Time Schedule, Publication, and Analysis Plans

Time Schedule

The time schedule for continuing the cross-site evaluation is summarized in Table 2. A 3-year clearance is requested for this project.

TABLE 2

Time Schedule

Receive OMB reapproval for study	April 2009
Continue data collection for centers funded in 2005, 2007, and 2008	Ongoing
Process and analyze data	Ongoing
Complete data collection for centers funded in 2005	September 30, 2009
Produce annual report under existing evaluation contract	September 30, 2009, and annually thereafter
Produce final report under existing evaluation contract	September 30, 2009
Begin data collection for new centers	October 1, 2009, and potentially annually thereafter
Complete data collection for centers funded in 2007	September 30, 2011
Complete data collection for centers funded in 2008	September 30, 2012

Publication Plan

A final report will be submitted to SAMHSA with anticipated subsequent dissemination to other interested parties, such as researchers, policymakers, and program administrators at the Federal, State, and local levels. Although not required under contract, it is also anticipated that results from this data collection will be published and disseminated in peer-reviewed publications. Examples of journals that may be considered as vehicles for publication include the following:

American Journal of Public Health	Journal of Clinical Child & Adolescent Psychology
American Psychologist	Journal of Consulting & Clinical Psychology
Child Abuse & Neglect	Journal of Emotional & Behavioral Disorders
Child Development	Journal of Health & Social Behavior
Child Maltreatment	Journal of Mental Health Administration
Children Today	Journal of School Psychology
Developmental Psychology	Journal of the American Academy of Child & Adolescent Psychiatry
Development & Psychopathology	Journal of Traumatic Stress
Evaluation Quarterly	Milbank Memorial Fund Quarterly
Evaluation Review	Social Services Review
Journal of Behavioral Health Services Research	Trauma Violence and Abuse
Journal of Child & Family Studies

Data Analysis Plan

All of the data collection and analytic strategies detailed in this package are linked to the evaluation questions. These linkages are shown in Table 3.

TABLE 3