Supporting Statement_5_26_2010 (3)

SUPPORTING STATEMENT FOR

Cross-Site Evaluation of Children’s Bureau’s Child Welfare Technical Assistance Implementation Centers and National Child Welfare Resource Centers

Submitted by

Department of Health & Human Services

Children’s Bureau

Washington, DC

Contact person:

Brian Deakins

Children’s Bureau

Administration on Children, Youth and Families

1250 Maryland Avenue, SW

Portals Building, Floor 8150

Washington, DC 20024

202-205-8769

Brian.Deakins@acf.hhs.gov

Revised May 26, 2010

Table of Contents

Section A: Justification 1

1. Circumstances making the collection of information necessary 1

2. Purpose and use of the information collection 1

3. Use of improved information technology and burden reduction 1

4. Efforts to identify duplication and use of similar information 1

5. Impact on small businesses or other small entities 1

6. Consequences of collecting the information less frequently 1

7. Special circumstances relating to the guidelines of 5 CFR 1320.5 1

8. Comments in response to the Federal Register Notice and efforts to consult outside the agency 1

9. Explanation of any payment or gift to respondents 1

10. Assurance of confidentiality provided to respondents 1

11. Justification for sensitive questions 1

12. Estimates of annualized burden hours and costs 1

13. Estimates of other total annual cost burden to respondents and record keepers 1

14. Annualized cost to the Federal government 1

15. Explanation for program changes or adjustments 1

16. Plans for tabulation and publication and project time schedule 5

17. Reason(s) display of OMB expiration data is inappropriate 1

18. Exceptions to Certification for Paperwork Reduction Act Submissions 1

Section B: Collections of Information Employing Statistical Methods 1

1. Respondent universe and sampling methods 1

a. Agency Results Survey 1

b. Training and Technical Assistance (T/TA) Activity Survey 1

c. Web-Based Network Survey 1

2. Procedures for the collection of information. 1

3. Methods to maximize response rates and deal with nonresponse 1

4. Tests of procedures or methods to be undertaken 1

5. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data 1

Appendix A: Cross-site evaluation research questions 1

Appendix B: Stakeholder Feedback 14

Appendix C: Sample reports 18

Section A: Justification

1. Circumstances making the collection of information necessary

The Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Youth and Families (ACF), U.S. Department of Health and Human Services (DHHS) seeks approval for the data collection instruments and procedures described herein. The proposed information collection is necessary to track, monitor, and evaluate the activities of Federally-funded training and technical assistance providers that deliver services to State, local, Tribal, and other publicly administered or publicly supported child welfare agencies and family and juvenile courts.

CB funds a network of training and technical assistance (T/TA) providers whose collective goal is to build the capacities of State, local, Tribal child welfare systems to successfully undertake practice, organizational, and systemic reforms necessary to achieve better outcomes for the children, youth and families they serve. The Child Welfare Training and Technical Assistance Network (T/TA Network) is comprised of 30 providers, including 11 National Child Welfare Resource Centers (NRCs) and five regional Child Welfare Implementation Centers (ICs). CB intends for information collection to begin upon OMB approval and to continue throughout the 5-year performance periods of its NRCs and ICs.

Legislative Background and Purpose

Two sections of Federal legislation provide further support and justification for this data collection. The proposed information collection is necessary to support CB’s ability to satisfy the general purposes of Section 203 of Adoption Opportunities [42 USC 5113(a)] as amended. Data will be used by CB to inform administration, coordination, and planning related to departmental activities affecting adoption and foster care, and in particular, decision making about the provision and appropriateness of its technical assistance. In addition, many of the ICs and NRCs that are the focus of the cross-site evaluation are discretionary grant projects authorized by Section 105 of the Child Abuse Prevention and Treatment Act [42 USC 5106] as amended. All grants for projects funded under this section are required to be evaluated for their effectiveness (Section 105(c)).

2. Purpose and use of the information collection

The information collection described in this request will allow CB to monitor the quality of services provided by NRCs and ICs to child welfare agencies and courts, to track changes in utilization of services, and to assess the effectiveness of the services delivered. The data will also help CB evaluate the collaboration and operation of the larger T/TA Network. Findings based on the information collected will facilitate monitoring of services provided by CB-funded discretionary grant recipients, coordination of federal consultation with CB-funded T/TA to States and Tribes, and State and Tribal accessibility to services. Findings will also be used to improve service coordination and delivery, prevent duplication, appropriately allocate resources, and increase the likelihood that the child welfare agencies and courts perform better as a result of the services received.

Data collected will address several critical evaluation questions. The first set of questions pertains specifically to IC and NRC T/TA activities and their relationship to intended organizational and systems changes within State and Tribal child welfare. The second set of questions pertains to the identity, cohesion and functioning of the larger T/TA Network. Research questions are provided in Appendix A. There are four primary data collection instruments employed in the cross-site evaluation: the Agency Results Survey, the T/TA Activity Survey, the Web-Based Network Survey and the OneNet technical assistance tracking system.

3. Use of improved information technology and burden reduction

The OneNet technical assistance tracking system has been designed to minimize respondent reporting burden by including an efficient set of data fields, and incorporating data-entry time saving features, such as drop-down fields, check boxes, and auto-fill features. In addition the system will be structured so that only one entry is necessary when multiple T/TA providers (ICs and/or NRCs) are working together to provide T/TA. Users will have 24-hour access to OneNet via secure internet connection. In addition, administration of the T/TA Activity Survey and the Web-Based Network Survey will be primarily electronic, utilizing email notification and Internet-based survey technologies. Using email address information from OneNet and from the T/TA Network directory, the national cross-site evaluators will send an email notification to respondents and invite them to complete the appropriate survey instrument by accessing a web-link to an online version of the survey. The majority of respondents are expected to have the capabilities to access the web-link to the surveys.

4. Efforts to identify duplication and use of similar information

The proposed instruments are intended to uniformly collect data that will allow CB to evaluate the collective efforts and effectiveness of the NRCs and ICs on child welfare systems. In addition to the proposed cross-site information collection, each NRC and IC will also design and conduct its own center-specific evaluation activities to address its unique legislative and programmatic priorities. CB has required its cross-site and center-specific evaluators to ensure data collection is necessary and complementary. The information collection and the center-specific evaluation activities of the NRCs and ICs have been coordinated to avoid potential duplication of data collection and reduce burden to respondents. The evaluators of each IC and NRC have met with the national cross-site evaluator and have reviewed each of the cross-site data collection instruments. The instruments have been revised to address potential overlap and the timing of data collection activities is being closely coordinated to minimize burden. While center-specific data will yield important and relevant information that will assist each NRC and IC to tailor and improve its services, center-specific data will not be sufficient to meet the cross-site purposes for the proposed information collection. There is no similar information available that addresses the coordination of T/TA activities and the functioning of the T/TA network and its 30 members.

5. Impact on small businesses or other small entities

The full range of information will be requested of all respondents. Organizational size will not affect the relevance of particular questions. A number of efforts are in place to minimize respondent burden, regardless of organizational size, for each of the data collection strategies described herein. Skip patterns have been included in the survey instruments based on the types of services received, and the timing of data collection activities is being coordinated to minimize respondent burden. Information being requested has been held to the absolute minimum and is sufficient enough to respond to the intended evaluation questions.

6. Consequences of collecting the information less frequently

In order to improve T/TA Network services and collaborate effectively to provide coordinated support to State and Tribal child welfare systems, CB and its providers need timely data on the provision of T/TA, the accessibility of services, the perceived effect and quality of the services received, and the interactions of service providers with one another. Less frequent data collection would inhibit the timely use of the information by the CB and providers to improve T/TA coordination and service quality.

7. Special circumstances relating to the guidelines of 5 CFR 1320.5

There are no special circumstances associated with this data collection.

8. Comments in response to the Federal Register Notice and efforts to consult outside the agency

Following publication of the notice that appeared in the Federal Register, Volume 74, Number 183, Wednesday, September 23, 2009, page 48568, no requests were received from the general public for copies of the proposed information collection instruments.

However, numerous opportunities were provided for direct stakeholders to review the proposed instruments and to contribute to their development throughout the design phase of this study. Appendix B identifies individuals who reviewed and offered written comments on the data collection approach, survey instruments (including scales and open-ended questions), and required elements and data entry forms for OneNet. The evaluation team has made an effort to be responsive to stakeholders’ comments whenever possible and has used their feedback in revising the data collection instruments.

The Agency Results Survey, the T/TA Activity Survey, and the Web-Based Network Survey were pilot-tested with individuals who were knowledgeable of the topics addressed and who had served in positions similar to the potential respondents (i.e, State/Tribal Child Welfare Directors, State technical assistance liaisons, current and former T/TA Network members). OneNet data collection forms were pilot-tested with individuals that were experienced with T/TA service provision and who will be future users of the OneNet system. Further information on the pilot tests is noted in Section B.

Following publication of the Federal Register Notice, several revisions were made to instruments based on comments from stakeholders and pilot testers. The Agency Results Survey was revised in response to stakeholder requests to be more inclusive of systems changes that may be underway in the Courts. The sampling and administration plans for the T/TA Activity Survey were adjusted in response to stakeholder concerns about respondent recall. The resulting burden figure—160 respondents with three responses per respondent each year—is a more accurate estimate of the number of respondents expected to complete multiple surveys. Two of the OneNet forms, the IC General TA Event Form and the NRC General TA Event Form, were consolidated into one form called the General T/TA Event Form, and the NRC Intake Form was deleted to help streamline data collection and eliminate duplication. In addition, several dropdown options and tabs were added to the OneNet forms in response to requests to better organize information and reduce the number of narrative fields users need to enter. These features will make it easier for respondents to fill in the forms. None of these revisions have added to the burden of completing the instruments and forms.

One comment received after the publication of the September 23, 2009 Federal Register notice was a request that the Agency Results Survey be administered to all State Court Improvement Program (CIP) Coordinators in order to capture a broader perspective on possible systems change underway in the Courts. This change in procedure could not be accommodated because of the significant resources that would be required to administer this additional data collection effort and to analyze and report the results.

9. Explanation of any payment or gift to respondents

No payments or gifts are provided to respondents for completing this information request.

10. Assurance of confidentiality provided to respondents

No assurances of complete confidentiality are provided to respondents. However, all respondents are informed of the importance of maintaining their privacy and that reported data are aggregated; they are not attributed to individuals.

11. Justification for sensitive questions

There are no questions or requirements of a sensitive nature contained in the survey instruments and data collection forms described herein.

12. Estimates of annualized burden hours and costs

Having applied hourly wage estimates to burden hours in each respondent category, the current annual cost to the respondents is as follows: (1) $5,707.62 for the Agency Results Survey; (2) $5,413.20 for the T/TA Activity Survey; (3) $389.33 for the Web-Based Network Survey; and

(4) $100,374.21 for OneNet. ¹ The total annual cost to the respondents if all data collection instruments were employed in the same given year is $111,884.36.

Exhibit A-12. Annual burden estimates

Instrument	Number of respondents	Number of responses per respondent	Average burden hours per response	Total burden hours
Agency Results Survey	74 State/Tribal Child Welfare Directors	1	1	74.00
Training and Technical Assistance (T/TA) Activity Survey	160 State/Tribal T/TA Recipients	3	0.25	120.00
Web-Based Network Survey	30 T/TA Network members	1	0.25	7.50
OneNet Forms
Implementation Project Information Form	5 ICs	5.40	0.5	13.50
Implementation Project T/TA Activity Form	5 ICs	280.80	0.33	463.32
Implementation Project Monthly Report Form	5 ICs	62.40	0.17	53.04
NRC T/TA Work Plan Form	11 NRCs	45	0.20	99.00
NRC T/TA Close-Out Form	11 NRCs	45	0.08	39.60
NRC T/TA Activity Form	11 NRCs	528	0.20	1161.60
General T/TA Event Form	11 NRCs + 5 ICs	26	0.25	104.00
Total				2135.12

13. Estimates of other total annual cost burden to respondents and record keepers

No additional cost burden will apply for respondents or record keepers.

14. Annualized cost to the Federal government

The associated costs for administering the surveys and developing and maintaining OneNet are outlined in Table A-14 below.  The annual cost to the Federal government to administer the Agency Results Survey is $8,838.93 (Years 2010, 2011, 2013); the annual cost to administer the T/TA Activity Survey is $2,287.32; and the annual cost (in Years 2010 and 2012) to administer the Web-Based Network Survey is $255.56.  The total annualized cost to develop the OneNet system is $50,400; the annualized cost to maintain OneNet (including labor and server costs) is $52,800.

Exhibit A-14. Annualized Costs for Survey and OneNet Administration

Instrument	Administration Activities	Staff Time	Total Cost
Agency Results Survey	74 State/Tribal CW Director Interviews	1.5	$8,838.93
T/TA Activity Survey	Sample TA events Distribute electronic survey & follow up	10.0 8.0	$1776.20 $511.12
Web-Based Network Survey	Distribute electronic survey & follow up	2.0	$255.56
OneNet	Development Maintenance	63.0 62.0	$50,400.00  $52,800.00
Total			$114,581.81

15. Explanation for program changes or adjustments

Not applicable.

16. Plans for tabulation and publication and project time schedule

Tabulation: Frequency distributions will be calculated to generate summaries of survey items and OneNet outputs, as well as to examine variability in the data. Parameter estimates, such as variances and means, will be established for each quantitative item. Cross-tabulations and significance tests will be conducted as appropriate. Content analysis will be conducted on open-ended survey items, and will entail systematic coding, creation of a hierarchy of codes, and cross-case and cross-source thematic analysis. Sample reports for the surveys are provided in Appendix C. Analyses will be conducted to determine subgroup variation. Multivariate analyses and group comparisons will examine the cumulative results of T/TA. Mixed categorical and interval variables will represent a range of measurements (i.e., characteristics of T/TA models; types and levels of intensity of T/TA support delivered to States and Tribes; and associated child welfare practice-, organizational- and system-level changes). An evaluation goal is to develop predictive models that identify factors associated with stronger and weaker T/TA results among States and Tribes.

Publication: The findings from the annual information collections will be summarized and tabulated in a series of annual briefings and reports to the CB beginning in the first year of data collection in FY 2010. Reports of research findings will include descriptive analyses, identification of moderating and mediating variables, and the implications of the findings. In the last stage of analysis beginning in 2013, data will be merged from multiple sources to enable final summative analyses to address major questions on the cumulative, overall results of IC and NRC T/TA. A final synthesis report of the project’s findings for all years will be submitted to the CB in FY 2013 for dissemination to federal, state and tribal stakeholders.

Project Timetable: The Agency Results Survey will be administered once per year in FY 2010, 2011, and 2013. The T/TA Activity Survey will be administered every three months from FY2011 through FY 2013. The Web-Based Network Survey will be administered twice, in years late FY 2010 and 2012. NRC and IC program staff will record basic information about their service delivery into OneNet within 10 business days of performing any substantial T/TA activity. Data entry will be ongoing throughout the providers’ respective project periods, which concludes in FY 2013.

17. Reason(s) display of OMB expiration data is inappropriate

The OMB expiration date for the information collection will appear on the instruments.

17. Exceptions to Certification for Paperwork Reduction Act Submissions

No exception is requested to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.

Section B: Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods ²

Agency Results Survey: The potential respondent universe for the Agency Results Survey includes Child Welfare Directors (or their designee) from all 50 States, the District of Columbia and Puerto Rico, and the Directors from 148 Tribes and 3 Territories (American Samoa, Guam, and the U.S. Virgin Islands) that currently receive Title IV-B funding and are eligible to receive T/TA from the ICs or NRCs. Nonprobability sampling strategies will be used. Two samples will be drawn. A census sample of Child Welfare Directors (or their designees) from the 50 States, the District of Columbia and Commonwealth of Puerto Rico will be included. This sample comprises all agencies that are federally monitored through the Child and Family Services Review and entitled to receive T/TA services from the CB. A census is necessary to obtain all 52 child welfare leaders’ perceptions and perspectives about T/TA utilization and services received with regard to specific change initiatives in their respective child welfare systems.

A purposive sample of 22 Child Welfare/Social Service Directors from the 148 Tribes and 3 Territories receiving Title IV-B funding will be selected. These Tribes, Tribal Consortia, and Territories will represent diversity in a number of areas: (1) Selection or approval for Children’s Bureau Implementation Projects in FY 2009; (2) Geographical representation across ACF Regions I-X; and (3) Amount of Title IV-B award (range is from $10,000 to $ 1 million).

Expert sampling will be used to identify the Indian and Territorial Child Welfare/Social Service Directors. ³ Selection of the Title IV-B sample will occur in conjunction with Children’s Bureau officials and ACF Regional Office staff. Any Tribal or Territorial community that declines participation in the study will be replaced with another Tribe or Territory that preserves the intended representative diversity.

A total of 74 State, Territorial, and Tribal Directors will be selected for survey administration. The baseline survey will be administered in FY 2010. Follow-up surveys will be administered in FY 2011 and 2013. There has been no previous collection.

Training and Technical Assistance (T/TA) Activity Survey: The target population for the T/TA Activity Survey consists of all State/Tribal recipients of T/TA delivered by the ICs or NRCs during FY 2010 through FY 2013. To represent the target population, we will construct a sampling frame from the T/TA tracking system, OneNet. The frame will consist of electronic forms that record relevant program information. Each form will represent one or more T/TA activities per day or multi-day on-site activity and one of the activities will be designated as the primary activity. Because we only plan to collect information about the primary activity, there is a one-to-one correspondence between the form and the primary activity. Therefore, the frame of forms is equivalent to a frame of primary activities. The following information will be available for each form on the frame: type of provider (NRC or IC), mode of T/TA (on-site or other), recipient of T/TA (State or Tribe), and other information that may be used to contact the recipient once sampled.

We propose a stratified sample design, with four strata defined by the cross of two dichotomous variables: (1) Type of provider: IC vs. NRC; and (2) Mode of T/TA: On-site vs. Other. Sample selection and data collection will take place at intervals of three months from FY2010 through FY2013; the sampling frame for each sample will consist of eligible forms within the period covered by the sample.

Under the stratified sample design, the sample of forms will be selected independently from each of the four strata. Within each stratum, we plan to select the sample systematically after the stratum frame is sorted by relevant recipient characteristics such as state, tribe, and possibly others, and by provider characteristics (e.g., which NRC or IC). The sorting imposes an implicit stratification within each stratum.

We propose to sample a total of 600 forms over a 12-month period. We will select 150 forms per three month period; equally divided among the four strata. Assuming a response rate of 80 percent, we expect to receive about 480 completed surveys (i.e., 3 surveys per the 160 respondents) per year out of the 600 selected for the sample. Exhibit B-1 shows the sample allocation and the expected number of completed surveys across the strata.

Exhibit B-1. Sample allocation over the strata per 12 month period

Stratum		Sample Size	Expected Number of Complete
NRC	On-Site	150	120
	Other Direct Consultation	150	120
IC	On-Site	150	120
	Other Direct Consultation	150	120
Total		600	480

Web-Based Network Survey: The potential respondent universe for the Web-Based Network Survey is 30 individuals who serve as the Directors of the provider organizations that comprise the Children’s Bureau T/TA Network. A nonprobability sampling strategy will be used. A census will be conducted with regard to this population. The expected response rate is 100 percent. The baseline survey will be administered in FY 2010 and a follow-up survey will be administered in FY 2012. There has been no previous collection.

2. Procedures for the collection of information.

No statistical methodology for stratification and sample selection will be used for the Agency Results Survey and the Web-Based Network Survey.

For the T/TA Activity Survey, we assumed that the marginal comparisons between NRC and IC activities and between On-site and Other activities are of primary analytical interest. Therefore, we will estimate the sample size in the context of two-sample comparisons. We calculated the sample size needed to detect a difference of 12 percent between two proportion estimates with 80 percent power at the 95 percent confidence level. Since the difference could be in either direction, a two-tailed test is appropriate. Assuming the proportion is centered around 70 percent in both populations, the number of respondents needed is about 240 per group (NRC vs. IC, or On-site vs. Other) for a total of 480. The following formula was used to derive the sample size.

where

is the sample size required per group to achieve the desired statistical power;

are the normal abscissas that correspond to the respective probabilities;

are the two proportions in the two-sample test;

is the simple average of and and .

3. Methods to maximize response rates and deal with nonresponse

Maximizing response rates is critical to the administration of the aforementioned surveys.⁴ Calculation of the response rates is as follows:

Exhibit B-3. Calculation of response rates

Survey	Respondent	# Respondents/ # Sampled	Response Rate (%)
Agency Results Survey	State Child Welfare Directors	52/52	100
	Child Welfare/Social Service Directors from Tribes, Tribal Consortia, and Territories (Title IV-B grantee)s	22/22	100
T/TA Activity Survey	State and tribal T/TA recipients	480/600	80
Web-Based Network Survey	T/TA Network Directors	30/30	100

The content and format of the three survey instruments were developed in close consultation with key stakeholders, including the IC and NRCs. In addition, development of the Agency Results Survey and T/TA Activity Survey was informed by previously developed measures involving T/TA provision.

Strategies that emphasize flexibility, confidentiality, and a respect for the respondent’s time facilitate timely participation. The following strategies will be implemented to maximize participation in the data collection and achieve the desired response rates: ⁵

1. Introduction and notification: An introductory letter will be sent on CB letterhead to inform all State and Tribal respondents about the administration of the Agency Results Survey. A description of the cross-site evaluation will be included in this mailing. For prospective Tribal respondents, this letter of introduction will also be sent to the Tribal Leader or Chairperson. Follow-up introductory calls will be made with prospective Tribal respondents to introduce the evaluation team and to address any questions about the data collection. In the subsequent years, reminder emails will be sent or telephone calls will be made to all State and Tribal respondents. We recognize that there may be some turnover in leadership over time and that we may have to re-introduce the survey. The CB will notify potential respondents about administration of the T/TA Activity Survey through listserv announcements. Similarly, the CB will notify the T/TA Network Directors about the administration of the Web-Based Network Survey through a listserv announcement and annual meetings of this group.

2. Timing of data collection: Discussions were held with stakeholders to determine optimal periods for data collection in order to minimize respondent burden and to facilitate recall. The Agency Results Survey will be conducted during the months of April-June in FYs 2010, 2011, and 2013. Administration will be coordinated with data collections efforts conducted by the ICs. The T/TA Activity Survey will be administered quarterly in FYs 2011-2013 (November-December and May-June each fiscal year). Data collection will occur within a three-month window following T/TA provision to facilitate respondent recall. In order to maximize participation and ensure a timely response, the Web-Based Network Survey will be administered within two weeks following the annual T/TA Network meetings in FY 2010 and 2012.

3. Pre-interview preparation: A copy of the Agency Results Survey will be sent to respondents in advance of the telephone interview. Background information for certain survey items will be “pre-filled” using information obtained from OneNet, semi-annual reports, or agency websites. Prior interviewer knowledge or familiarity with each State or Tribe’s child welfare system will expedite administration of the interview. Pre-interview preparation is not applicable to the T/TA Activity Survey and Web-Based Network Survey.

4. Administration: For the Agency Results Survey, the telephone interviews will be scheduled at the respondents’ convenience. The evaluation team will confirm the interview 2-3 days beforehand and re-schedule any interviews as necessary to accommodate any changes in a Director’s schedule, given the dynamic work environment of public child welfare agencies. ⁶ Similarly, the evaluation team will do the same with Tribal respondents, given the potential for schedule changes due to community obligations or seasonal fluctuations in cultural activities. ⁷ (Running Wolf, Soler, Manteuffel, Sondheimer, Santiago & Erickson, 2004). For the T/TA Activity Survey and Web-Based Network Survey, an email notification will be sent to all sampled T/TA recipients with a request to complete the survey (i.e., by accessing a web-link to an online version of the survey or accessing an attached survey to complete and return via email, mail or secure fax).  Electronic participation will allow respondents the flexibility to complete the survey at the most convenient time with minimal burden. Approximately two weeks after sending this initial email, a reminder email will be sent to those respondents who have not yet completed the survey. Two weeks after sending this reminder email, the evaluation team will call non-respondents to remind them of the survey. During these follow-up calls, evaluation staff will administer the survey by telephone if agreed to by the respondent.

5. Alternate response methods: All respondents will be given the option to use an alternate method. For the Agency Results Survey, if a respondent prefers to submit written responses to the survey in lieu of participating in a telephone interview, we will provide him/her with a paper version to complete via fax, email, or mail. Similarly, paper versions of the T/TA Activity Survey and Web-Based Network Survey will be sent to respondents upon request. Alternately, the latter surveys will be administered through a telephone interview if requested to accommodate any special needs.

6. Assurances of data confidentiality: Respondents to all surveys will be assured that reported data are aggregated and not attributable to individuals or organizational entities.

There are no incentives provided for participation in any of the surveys.

3. Tests of procedures or methods to be undertaken

The four instruments contained herein were subject to review and feedback by key stakeholders, including the CB, ACF Regional Office staff, ICs, and, NRCs, and the T/TA Network.

The Agency Results Survey, T/TA Activity Survey, and Web-Based Network Survey were pilot tested to confirm survey item validity and to identify possible procedural or methodological challenges in need of attention or improvement. Pilot tests were conducted for each instrument using a sample of no more than nine respondents (i.e., former State and Tribal child welfare Directors, former child welfare agency personnel, and previous National Resource Center Directors). Following the pilot tests, the instruments were refined to minimize burden and improve utility. Similarly, a paper version of the web-based forms used in the OneNet tracking system was tested by prospective users and refined. The pilot tests were instrumental in determining the amount of time required to complete the surveys and forms and develop the burden estimates.

User access and responsiveness to the web-based methodology for completing the T/TA Activity Survey, Web-Based Network Survey, and OneNet forms will be pilot-tested. Should this method require revision, modifications will be made prior to implementation of the full data collection.

3. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data

National Cross-Site Evaluation Contractor		Statistical Consultant:
James Bell Associates 1001 19th Street North, Suite 1500 Arlington, VA (703) 528-3230	ICF International 10503 Rosehaven Street Suite 400 Fairfax, VA 22030	Y. Michael Yang, Ph.D. Chief Statistician ICF International 9300 Lee Highway Fairfax, VA 22031 (703) 934-3320

Appendices

Appendix A: Cross-site evaluation research questions

ICs, NRCs, and State/Tribal systems change

1a. Is utilization of T/TA increasing among those State and Tribal child welfare systems that have demonstrated the greatest needs?

1b. Is utilization of T/TA increasing among those State and Tribal child welfare systems

that have been specifically targeted for outreach by the ICs?

2. Have States and Tribes that have entered into formal partnerships with ICs achieved their desired outcomes?

3. To what degree are the T/TA activities and approaches of the ICs and NRCs resulting in changes in organizational culture/climate and capacity in States and Tribes?

4. To what degree are the T/TA activities and approaches of the ICs and NRCs resulting in desired systems change in State and Tribal child welfare systems?

5. What key variables are correlated with whether desired systems change is achieved as a result of T/TA provided by the ICs and NRCs?

6. Have intended systems changes also resulted in improved outcomes for children and families in State and Tribal child welfare systems?

7. What are the key factors that facilitate and impede utilization of the NRCs and ICs by State and Tribal child welfare systems?

8. What is the overall quality of the T/TA that is provided by the ICs and NRC? And how has the overall quality changed?

9. What is the nature and quality of the relationships and interactions established between States and Tribes (and other key constituents) and the ICs and NRCs with whom they work?

10. What are the costs of providing the type, frequency, and intensity of T/TA that is most likely to result in the desired systems change and improved outcomes?

Identity, Cohesion, and Functioning of the T/TA Network

11. What is the nature and quality of the relationships and interactions established between the members of the T/TA Network?

12. Are collaboration, coordination, and application of the Child and Family Services Review (CFSR) and Systems of Care (SOC) principles increasing among T/TA Network members over the life of the projects?

13. To what degree does the transfer of knowledge and information take place between T/TA Network members?

14. To what degree are members of the T/TA Network subscribing to and promoting a shared identity through common messaging in outreach and service delivery?

Appendix B: Stakeholder Feedback

Individuals and Organizations Providing Feedback on

Cross-Site Evaluation Data Collection Strategies

Appendix B

Individuals and Organizations Providing Feedback on

Cross-Site Evaluation Data Collection Strategies

Review and Comments on Data Collection Approach, Instruments, and Data Collection System (OneNet)
Implementation Center Representatives
Michelle I. Graef, Ph.D. Associate Director, Midwest Child Welfare Implementation Center Research Associate Professor UNL Center on Children, Families & the Law office:  402.472.3741  cell:  402.540.4791 toll-free:  888.523.8055	Sarah Kaye, Ph.D. Research Assistant Professor Ruth H. Young Center for Families and Children University of Maryland School of Social Work skaye@ssw.umaryland.edu
Cathy Sowell, LCSW Partner, Western and Pacific Child Welfare Implementation Center Department of Child & Family Studies Louis de la Parte Florida Mental Health Institute University of South Florida (813) 974-3628	Susan N. Kanak Project Director NCIC Muskie School of Public Service University of Southern Maine, skanak@usm.maine.edu
Kris Sahonchik National Resource Center for Organizational Improvement Director of Strategy and Coordination kriss@usm.maine.edu 207-780-5856	Tammy Richards NCIC, University of Southern Maine Muskie School of Public Service 207.780.5859 tammyr@usm.maine.edu
Julie R. Morales, Ph.D. Senior Research Associate Butler Institute for Families University of Denver phone: 303-871-4873 email: Julie.Morales@du.edu
National Resource Center Representatives
Peter Watson, Director National Child Welfare Resource Center for Organizational Improvement Muskie School of Public Service University of Southern Maine office: 207.228.8330 cell: 207.632.0892	Dorothy I Ansell National Child Welfare Resource Center for Youth Development University of Oklahoma 918-660-3700
National Resource Center Representatives (continued)
Scott Trowbridge, Esq. Staff Attorney American Bar Association Center on Children and the Law T: (202)662-1747 trowbris@staff.abanet.org	Gerald P. Mallon, DSW Professor and Executive Director National Resource Center for Family-Centered Practice and Permanency Planning Hunter College School of Social Work 212 452-7043/phone gmallon@hunter.cuny.edu
Debbie Milner NRC-CWDT Project Manager Child Welfare League of America home/office 850-622-1567 dmilner@cwla.org	Theresa Costello Director, National Resource Center for Child Protective Services Deputy Director, ACTION for Child Protection 505-345-2444 office 505-301-3105 mobile
Sharonlyn Harrison, Ph.D. Director Public Research and Evaluation Services (PRES) www.pres-online.com PH/866-945-1590
Other Network Members and Stakeholders
Linda Baker, Director FRIENDS National Resource Center for CBCAP (919) 768-0162	Anita P. Barbee, MSSW, Ph.D. Professor Kent School of Social Work University of Louisville (502) 852-0416 (502) 852-0422 anita.barbee@louisville.edu
Central and Regional Office Staff
Brian Deakins Capacity Building Division HHS/ACF/ACYF/Children's Bureau (202) 205-8769 Brian.Deakins@acf.hhs.gov	John L. (Jack) Denniston Child Welfare Program Specialist (contractor) Children's Bureau Division of Research and Innovation ICF International 919-968-0784 john.denniston@acf.hhs.gov
Central and Regional Office Staff (continued)
Sally Flanzer, Ph.D. Regional Program Manager Child Welfare Administration for Children and Families Region IX Ph: 415-437-8425 E-mail: Sally.Flanzer@acf.hhs.gov	Jesse Wolovoy Program Specialist Children's Bureau/Administration for Children and Families Phone: 215-861-4014
	Melissa Lim Brodowski, MSW, MPH Office on Child Abuse and Neglect Children's Bureau, ACYF, ACF, HHS Washington, DC 20024 phone:  202-205-2629 email: melissa.brodowski@acf.hhs.gov

Appendix C: Sample Reports

T/TA Activity Survey Sample Reports

1. Utilization of Children’s Bureau’s T/TA Providers

   1. Influence of federal, network, and State/tribal factors in requesting TA

Table I-1. Factors influencing the use of T/TA

	Total N	Mean Score a
Federal Factors
ACF Regional Office suggestion/referral
CFSR findings/PIP development
Federal law or policy change
Other Federal factors (Specify):
T/TA Network Factors
Outreach to your State/Tribe by the National Resource Center
Outreach to your State/Tribe by the Implementation Center in your ACF Region
Prior use of National Resource Center services
Prior use of Implementation Center services
Geographic proximity of the National Resource Centers
Geographic proximity of the Implementation Center in your ACF Region
Recommendation/Referral from other National Resource Centers
Recommendation/Referral from another Implementation Center (outside your ACF Region)
Peer networking activities facilitated by the National Resource Centers
Peer networking activities facilitated by the Implementation Centers in your ACF Region
Other T/TA Network factors (Specify):
State/Tribal Factors
Recommendation from other State/Tribe
Specific State/Tribal incident (e.g., child fatality)
State/Tribal quality assurance review
Agency/organization leadership
Lawsuit/legal settlement
State/local law or policy change
Other State/Tribal factors (Specify):

^a Based on a 5-point scale: 1 indicates no influence; 5 indicates a great deal of influence

2. Experience with Children’s Bureau’s NRCs and ICs

1. Experience seeking assistance from a specific NRC/IC (see Tables II-1 and 2)

2. Suggestions on improving the request and approval process

3. Experience working with IC and NRC (Activity specific) (see Table II-3)

4. Satisfaction with IC and NRC technical assistance (see Tables II-4 and 5)

5. Suggestions for improving technical assistance

Table II-1. Experience seeking assistance from Children’s Bureau’s NRCs

	Total N	Mean Score a
The process for requesting technical assistance was clear
Our State/Tribe knew whom to contact for technical assistance
Our State/Tribe received a timely response to our request for technical assistance

^a Based on a 5-point scale: 1 indicates strongly disagree; 5 indicates strongly agree

Table II-2. Experience seeking assistance from Children’s Bureau’s ICs

	Total N	Mean Score a
The process for applying for an Implementation Project (IP) was clear
The selection process for an IP was clear and transparent

^a Based on a 5-point scale: 1 indicates strongly disagree; 5 indicates strongly agree

Table II-3. Experience with IC and NRC technical assistance ^a

	Total N	Mean Score b
Knowledge and expertise of consultants that provided TA
The consultants were knowledgeable about the issues being addressed
The consultants effectively utilized the knowledge and expertise of our State/Tribe
The consultants were able to build a positive working relationship with our staff
The consultants effectively facilitated conversations with our staff
The consultants effectively facilitated the process and work necessary to address our need or problem
The consultants were able to understand the State/Tribe’s unique situation and tailor the technical assistance to our needs
Overall, the consultants were effective
Outcome of the T/TA event
The T/TA event addressed our State’s or Tribe’s needs
The [IC/NRC Name] provided our State or Tribe with a viable plan for implementing the recommended strategies
The T/TA event increased our State’s or Tribe’s knowledge
The T/TA event increased our State’s or Tribe’s skills
The T/TA event improved our State’s or Tribe’s service capacity
The T/TA event improved our State’s or Tribe’s ability to better serve children, youth, and families

^a These data will be reported in aggregate and individual tables will be created for the ICs and NRCs

^b Based on a 5-point scale: 1 indicates strongly disagree; 5 indicates strongly agree

Table II-4. Satisfaction with NRC technical assistance

Satisfaction with communication, information sharing, relationships, and follow through	Total N	Mean Score a
Our State/Tribe was satisfied with the level of accessibility of the NRC
The plan for T/TA was appropriate for achieving our State/Tribe’s objectives
Our State/Tribe was satisfied with the frequency of communication with NRC
Our State/Tribe felt comfortable disclosing our areas of concerns or weaknesses to NRC
Our State/Tribe played an active part in decision making regarding the course of action to be taken by NRC
Overall, our State or Tribe was satisfied with the relationship developed with NRC

^a Based on a 5-point scale: 1 indicates strongly disagree; 5 indicates strongly agree

Table II-5. Satisfaction with IC technical assistance

Satisfaction with communication, information sharing, relationships, and follow through	Total N	Mean Score a
Our State/Tribe was satisfied with the level of accessibility of the IC
The IC’s T/TA was timely in relation to the goals established in the work plan
The plan for T/TA was appropriate for achieving our State/Tribe’s objectives
Our State/Tribe was satisfied with the frequency of communication with the IC
Our State/Tribe felt comfortable disclosing our areas of concerns or weaknesses to the IC
There was flexibility in the Memorandum of Understanding to make any necessary modifications
Our State/Tribe played an active part in decision making regarding the course of action to be taken by the IC
Overall, our State or Tribe was satisfied with the relationship developed with the IC

^a Based on a 5-point scale: 1 indicates strongly disagree; 5 indicates strongly agree

3. Coordination of Multiple T/TA Providers

   1. Coordination of the technical assistance from multiple T/TA providers

   2. Suggestions for improving the coordination of technical assistance

Table III-1. Coordination of T/TA from multiple T/TA providers

	Total N	Mean Score a
T/TA providers (NRCs and ICs) structured their activities to avoid duplication
were knowledgeable of each other’s efforts
coordinated the dates of their on-site activities
Overall, the technical assistance providers coordinated their activities

^a Based on a 5-point scale: 1 indicates strongly disagree; 5 indicates strongly agree

4. State or Tribe’s Response to T/TA

   1. Perception of the State/Tribe’s response to TA provided

   2. Other comments on the State/Tribe’s response to T/TA

Table IV-1. State/Tribe’s response to T/TA

	Total N	Mean Score a
Most supervisors or frontline staff were receptive to recommendations of NRC/IC
middle managers were receptive to recommendations of NRC/IC
senior administrative/managerial staff were receptive to recommendations of NRC/IC
State/Tribe has staffing resources (both availability and expertise) to implement recommendations
financial resources to implement recommendations

^a Based on a 5-point scale: 1 indicates strongly disagree; 5 indicates strongly agree

5. Participant Characteristics

Note: Aggregated descriptive data based on drop-down fields and open-ended survey items E1-E5.

1. Types of positions held: All respondents

2. Organizational entities represented in survey: All types

3. Divisions/units represented in survey: All types

4. Average number of years in position: All respondents

5. Average number of years with the agency: All respondents

1 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed. In 2010, all data collection strategies except for the T/TA Activity Survey are included and the respondent burden and annualized cost are estimated to be 2015.12 hours and $106,471.16. In 2011 and 2013, the T/TA Activity Survey will be conducted but the Web-based Network Survey will not be administered, resulting in a net increase to the annual burden and cost of 2127.62 and $111,495.03 respectively. In 2012, the Agency Results Survey will not be administered and the annual burden and cost will be 2061.12 hours and $106,176.74.

2 Timely data entry in the OneNet system is part of the ICs’ and NRCs’ work responsibility and federal reporting requirements. OneNet will be used as the sampling frame to draw the sample of respondents for the T/TA Activity Survey and is addressed below.

3 Trochim, W.M.K. 2001. The Research Methods Knowledge Base. Cincinnati, OH: Atomic Dog Publishing.

4 This section does not address data collection for OneNet as data entry is part of the grantee’s work responsibility and federal reporting requirements.

5 Strategies that pertain to two or more data collections are discussed together.

6 Brooks, D. & Wind, L.H. (2002). Challenges implementing and evaluating child welfare demonstration projects. Children and Youth Services Review, 24, nos. 6/7, 379-383. Solomon, B. (2002). Accountability in public child welfare: Linking program theory, program specification and program evaluation. Children and Youth Services Review, 24, nos. 6/7, 385-407.

7 Running Wolf, P., Soler, R., Manteuffel, B., Sondheimer, D., Santiago, R.L., Erickson, J. (2004). Cultural Competence Approaches to Evaluation in Tribal Communities. Rockville, MD: Substance Abuse and Mental Health Services Administration.