Ssb Capp_2015-11-06

Permanency Innovations Initiative: Phase 4

OMB Information Collection Request

0970 - 0408

Supporting Statement

Part B

November 2015

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officer:

Maria Woolverton

B1. Respondent Universe and Sampling Methods

The participating sites for the CAPP study are Fresno, Santa Clara, Humboldt County child welfare agencies, and the Los Angeles County offices of Pomona and Wateridge.

Each site will identify and create a list of all parent-legal guardians and caregivers who meet the eligibility criteria to complete a one-time survey. Eligible study participants include parent-legal guardians or foster caregivers of children who are assigned to social workers trained to implement the CAPP Child and Family Practice Model; have been part of cases served by a CAPP-trained social worker for 90 days or more; can speak and read English or Spanish; and are age 18 or older. Parent-legal guardians are in Family Reunification (FR) status, where a child in foster care and the parents from whom the child was removed are receiving services to reunify as a family. Caregivers are foster care providers caring for children in Permanent Placement (PP) status, where the child in foster care is not expected to reunify with a parent or parent or legal guardian, and services are targeted toward their status in foster care and achieving adoption, guardianship or another planned permanent living arrangement.

Westat will prepare survey packets for all survey participants. Surveys for each parent-legal guardian and caregiver in the study will be assigned a Survey ID consisting of a 3-part code that identifies the site, caseworker, and parent-legal guardian or foster caregiver. The code will appear on all pages of the informed consent and the survey so Westat can match the informed consent and survey should pages become separated. This system also is meant to ensure that each parent-legal guardian and each caregiver completes no more than one survey. The Local Survey Coordinator in each CAPP site will maintain a crosswalk linking the Survey ID to the: 1) social worker, and 2) parent-legal guardian or caregiver participant served by the caseworker.

Parent-Legal Guardian Sample

The maximum total number of returned surveys for the combined parent-legal guardian and caregiver samples is estimated at 3,436 (1673 parent-legal guardian surveys and 1763 caregiver surveys) In order to maximize the response rate and also stay at or below our estimated burden threshold, the Los Angeles-Wateridge site will sample 50% of the eligible population by selecting one of the two offices that have FR and PP cases. Other CAPP sites will sample 100% of eligible parent-legal guardians. The estimated numbers of eligible parent-legal guardians in FR status for each CAPP site are: Fresno County = 531, Humboldt County = 115, Los Angeles-Pomona = 517, Los Angeles-Wateridge = 634, and Santa Clara County = 434. Westat will disseminate surveys to the entire parent-legal guardian sample beginning February 1, 2016.

Westat will send the designated number of prepared parent-legal guardian survey packets to each CAPP site Local Survey Coordinator. Sealed parent-legal guardian survey packets will contain the combined informed consent and questionnaire, and a pre-addressed, postage-paid return envelope. The Survey ID and name of the social worker will be on a sticker on the outside of parent-legal guardian survey packets so Local Survey Coordinators can provide surveys to the correct survey distributor.

The Local Survey Coordinator will provide survey packets to each selected CAPP-trained caseworker or other identified survey distributor who will hand-distribute survey packets to eligible parent-legal guardians during their next in-person contact. Since monthly contacts are required and each survey distributor will distribute a specified number of packets, it is anticipated that all surveys will be distributed within 2 to 4 weeks. The survey distributor will read the prepared script to the parent-legal guardian that briefly describes what is contained in the survey packet, instruct the parent-legal guardian how to complete and return the informed consent and survey, and that they may call Westat’s toll-free 800 number listed in the survey packet for questions and assistance completing the questionnaire or to take the survey by telephone. The script includes a list of potential questions from parent-legal guardians and acceptable answers the survey distributor can provide. See Attachment B2b: Parent-Legal Guardian Questionnaire Distributor Script and Attachment B2a: Parent-Legal Guardian Questionnaire Distributor Script Spanish.

Survey distributors will indicate which surveys have been distributed by removing the sticker from the outside of the survey packet and placing it on the Survey Distribution Tracking Sheet. They will also record the name of the parent-legal guardian and distribution date. Each CAPP pilot site Local Survey Coordinator will collect Survey Distribution Tracking Sheets, enter information on the electronic Master Survey Tracking Sheet, and upload electronic files onto Westat’s secure server. A month after the start of the data collection, survey distributors will hand out duplicate survey packets to parent-legal guardians who have not returned a completed questionnaire to Westat or completed the questionnaire by telephone.

Foster Caregiver Sample

Each site will send a CAPP Study Frequently Asked Questions (FAQ) letter by US Mail to all identified caregivers from the caseloads of the selected caseworkers to obtain passive consent for Westat to have access to their contact information. Each caregiver will receive the letter in English and Spanish. The letter will describe the study to eligible caregivers and indicate that their contact information will be shared with Westat for research purposes if they do not contact the county by a certain due date. See Attachment B1a: Study FAQ Letter for Caregivers 8-15 and Attachment B1b: Study FAQ Letter for Caregivers Spanish 8-15. If the caregiver agrees to share contact information by way of passive consent, the local child welfare agency will include them in the pool of eligible caregivers to participate in the study. If a caregiver contacts their local child welfare agency and indicates that they do not want to share their contact information with Westat, their denial of consent will be recorded by the child welfare agency and they will be excluded from the pool of eligible caregivers to participate in the study.

Westat will disseminate surveys to all eligible caregivers who passively consented starting March 1, 2016. The estimated number of caregivers caring for children in PP status for each site are: Fresno County = 711, Humboldt County = 121, Los Angeles-Pomona = 416, Los Angeles-Wateridge = 468, and Santa Clara County = 460. Each CAPP Local Survey Coordinator will compile mailing addresses and telephone numbers for the identified caregivers, along with the name of the social worker and child’s first name and last initial. If the social worker works with more than one PP child in the caregiver’s home, the list will contain the oldest Permanent Placement (PP) child. The Local Survey Coordinator will upload caregiver information to Westat’s secure server. Westat will use the mailing addresses to send follow-up postcards, duplicate survey packets, and if needed, contact caregivers via telephone.

Power Analysis

There are two populations to consider that include parent-legal guardians and caregivers. These are divided into two subgroups by race, the first comprising African Americans (AA) and Native Americans (AI) and the second group comprising all other (O) races. There is an understanding that the CAPP initiative will help close the gap between the two groups for both parent-legal guardians and caregivers. However, since there are no baseline measurements of the pre-CAPP intervention and no comparisons over time, we cannot directly test the intervention. With a single comparison between the groups at a single time point, we can only test whether there is evidence for the existence of a difference between the race groups at the point in time in question for the outcome measure under scrutiny. We also cannot directly relate the presence or absence of a difference to the CAPP treatment. The measures we will be assessing are intended to inform intermediate goals and long term goals. For example, the tests performed can be used to establish a baseline measure between racial groups which may establish the ability to help draw more complex inferences over time than are possible by these single tests. Moreover, the population is also spread over several counties. Therefore, some clustering at the caseworker level may exist, but this analysis does not adjust for clustering.

The two populations of parents-legal guardians and caregiver, are divided into the treatment group (comprising African Americans and Native Americans) and the control group (Whites and Other races). The treatment is aimed at reducing the disparities between the two groups for both parent-legal guardians and caregivers.

We examined the power to detect differences in measures between treatment and control groups for our proposed sampling plan. The assumption is that a disparity measure is in the form of a five point scale (such as, for example, 1=Very much so … 5=Not at all), and we are considering the mean measure in the treatment and control groups. We assume that the standard deviation is 1, which is equivalent to one category on the scale. This is a conservative assumption in that we assume the responses are fairly variable within each group, making it harder to detect differences between them. We calculated the power to detect various differences between the two group means for the given sample size and also the minimum detectable difference at the given sample size at 80% power. We have calculated the power assuming both a 60% response rate and a 75% response rate. We have budgeted for a 75% response rate in case of greater response than anticipated; however, based on the pilot study, we expect that a 60% response rate will be more likely. As evidenced by the power analyses, we will have the power to detect impacts at either 60% or 75% response rates.¹.

For the parent-legal guardian study, Table B1 presents the sample size by county:

Table B1. Parent-Legal Guardian Sample Sizes by County
	Fresno		Humboldt		LA - Pomona		LA - Wateridge		Santa Clara		Total Sample
	N	%	N	%	N	%	N	%	N	%	N
Treatment group	170	32.0%	49	42.6%	149	28.8%	189	29.9%	94	21.7%	651
Control group	361	68.0%	66	57.4%	368	71.2%	444	70.1%	340	78.3%	1579
Total	531	100%	115	100%	517	100%	633	100%	434	100%	2230

Table B2 presents the power calculations by county for the parent-legal guardian sample:

Table B2. Parent-Legal Guardian Power Calculations by County
	60% Response Rate			75% Response Rate
County	Difference in group means	Power to detect	Minimum detectable difference at 80% power	Difference in group means	Power to detect	Minimum detectable difference at 80% power
Fresno	0.5	99.4%	0.30	0.5	99.9%	0.27
	0.4	95.4%		0.4	98.1%
	0.3	80.2%		0.3	87.4%
	0.2	50.7%		0.2	58.6%
	0.1	20.8%		0.1	23.8%
Humboldt	0.5	65.1%	0.61	0.5	74.0%	0.54
	0.4	49.3%		0.4	57.3%
	0.3	33.5%		0.3	39.3%
	0.2	20.3%		0.2	23.3%
	0.1	10.8%		0.1	11.8%
LA - Pomona	0.5	99.0%	0.31	0.5	99.8%	0.22
	0.4	93.8%		0.4	97.3%
	0.3	77.0%		0.3	84.8%
	0.2	47.8%		0.2	55.5%
	0.1	19.8%		0.1	22.5%
LA - Wateridge	0.5	99.7%	0.28	0.5	>99.9%	0.25
	0.4	97.2%		0.4	99.0%
	0.3	84.7%		0.3	91.1%
	0.2	55.3%		0.2	63.6%
	0.1	22.5%		0.1	25.8%
Santa Clara	0.5	95.2%	0.38	0.5	98.1%	0.33
	0.4	84.1%		0.4	90.8%
	0.3	63.3%		0.3	72.1%
	0.2	37.3%		0.2	43.7%
	0.1	16.3%		0.1	18.4%

As can be seen, we have the least power in Humboldt, which has the smallest population, but even there, we can detect a difference of 0.61 or 0.54, less than one unit on the scale, at 80% power.

For the caregiver study, Table B3 presents the sample size by county:

Table B3. Caregiver Sample Sizes by County
	Fresno		Humboldt		LA - Pomona		LA - Wateridge		Santa Clara		Total Sample
	N	%	N	%	N	%	N	%	N	%
Treatment group	250	35.2%	52	43.0%	141	33.9%	163	34.8%	84	18.3%	690
Control group	461	64.8%	69	57.0%	275	66.1%	305	65.2%	376	81.7%	1486
Total	711	100%	121	100%	416	100%	468	100%	460	100%	2176

Table B4 presents the power calculations by county for the caregiver sample:

Table B4. Caregiver Power Calculations by County
	60% Response Rate			75% Response Rate
County	Difference in group means	Power to detect	Minimum detectable difference at 80% power	Difference in group means	Power to detect	Minimum detectable difference at 80% power
Fresno	0.5	>99.99%	0.25	0.5	>99.99%	0.23
	0.4	98.9%		0.4	99.7%
	0.3	90.5%		0.3	95.2%
	0.2	62.7%		0.2	71.2%
	0.1	25.5%		0.1	29.4%
Humboldt	0.5	67.4%	0.59	0.5	75.7%	0.53
	0.4	51.2%		0.4	59.1%
	0.3	34.9%		0.3	40.5%
	0.2	21.0%		0.2	24.0%
	0.1	11.0%		0.1	12.0%
LA - Pomona	0.5	98.2%	0.33	0.5	99.4%	0.30
	0.4	91.0%		0.4	95.5%
	0.3	72.4%		0.3	80.5%
	0.2	44.0%		0.2	51.0%
	0.1	18.5%		0.1	20.9%
LA - Wateridge	0.5	99.0%	0.31	0.5	99.8%	0.28
	0.4	93.9%		0.4	97.2%
	0.3	77.2%		0.3	84.8%
	0.2	48.0%		0.2	55.4%
	0.1	19.8%		0.1	22.5%
Santa Clara	0.5	93.9%	0.39	0.5	97.4%	0.35
	0.4	81.8%		0.4	88.9%
	0.3	60.6%		0.3	69.3%
	0.2	35.6%		0.2	41.6%
	0.1	15.7%		0.1	17.7%

B2. Procedures for Collection of Information

Parent-Legal Guardian Survey Administration

Westat will send the designated number of prepared parent-legal guardian survey packets to each CAPP site Local Survey Coordinator. Sealed parent-legal guardian survey packets will contain the combined informed consent and survey questionnaire in English and Spanish, and a pre-addressed, postage-paid return envelope. The Survey ID and name of the social worker will be on a sticker on the outside of parent-legal guardian survey packets so Local Survey Coordinators can provide surveys to the correct survey distributor.

The Local Survey Coordinator will provide survey packets to each selected CAPP-trained caseworker or other identified survey distributor who will hand-distribute survey packets to eligible parent-legal guardians. The survey packets will be handed to eligible parent-legal guardians as they are met by the survey distributor during the next in-person contact. The survey distributor will read the prepared script to the parent-legal guardian that briefly describes what is contained in the survey packet, instruct the parent-legal guardian how to complete and return the informed consent and survey, and that they may call Westat’s toll-free 800 number listed in the survey packet for questions and assistance completing the survey or to take the survey by telephone. The script includes a list of potential questions from parent-legal guardians and acceptable answers the survey distributor can provide. See Attachment B2a: Parent-Legal Guardian Questionnaire Distributor Script 8-15 and Attachment B2b: Parent-Legal Guardian Questionnaire Distributor Script Spanish 8-15.

Survey distributors will indicate which surveys have been distributed by removing the sticker from the outside of the survey packet and placing it on the Survey Distribution Tracking Sheet. They will record the name of the parent-legal guardian and distribution date. Survey distributors will also record the names of who they have approached who did not want to complete the survey in order to better understand rates of refusal. Each CAPP site Local Survey Coordinator will collect Survey Distribution Tracking Sheets, enter information on the electronic Master Survey Tracking Sheet, and upload electronic files onto Westat’s secure server. Westat will track surveys disseminated compared to surveys received or administered via telephone. As needed, Westat will provide two sets of duplicate packets to Local Survey Coordinators for survey distributors to disseminate to respondents who did not mail back the original questionnaire. Duplicate packets will have the same codes as the original packets.

The informed consent includes instructions for each parent-legal guardian to indicate a name and address to mail a token of appreciation consisting of a $50.00 Visa gift card. Westat will mail the token of appreciation upon receipt of the paper-and-pencil survey questionnaire or administration of the questionnaire via telephone.

Caregiver Survey Administration

Based upon CAPP’s experience in the pilot study and subsequent IRB approval, Westat will prepare the initial mailing of caregiver survey packets. Caregiver survey packets will contain the combined informed consent and questionnaire in English and Spanish, and a pre-addressed, postage-paid return envelope. Caregiver questionnaires will have a space for Westat to place a sticker naming the social worker from the selected case and the PP child they work with in the caregiver’s home. If the social worker works with more than one PP child in the caregiver’s home, the sticker will name the oldest PP child. This includes placing the sticker naming the social worker from whom they selected the case and the PP child they work with in the caregiver’s home. Names on stickers will be taken from the contact information the Local Survey Coordinator uploads to the Westat secure site.

As with the parent-legal guardian survey, the informed consent instructs caregivers to call Westat’s toll-free 1-800 number listed on the informed consent form in the survey packet for questions and assistance completing the survey or to take the survey by telephone.

Westat will track surveys disseminated compared to surveys received and completed via telephone. As part of caregiver identification, each site will have compiled mailing addresses and telephone numbers for each of their identified caregivers, pending passive consent, and upload contact information to Westat’s secure server. Approximately two weeks after initial mailing, Westat will send a reminder postcard to all study-eligible caregivers at their mailing address. See Attachment B3: Caregiver Reminder Postcard English-Spanish 8-15 for an example of postcard text. Approximately four weeks after the initial mailing, Westat will mail duplicate surveys to study caregivers who did not return the initial survey. Approximately six weeks after the initial mailing, if a sufficient number of questionnaires have not been received to conduct rigorous analyses, Westat will conduct a final round of telephone reminders to study caregivers who did not respond to the survey. The telephone reminder will include an option to administer the survey by telephone at their convenience.

The informed consent includes instructions for each caregiver to indicate a name and address to mail a token of appreciation consisting of a $50.00 Visa gift card. Westat will mail the token of appreciation upon receipt of the paper-and-pencil survey questionnaire or administration of the questionnaire via telephone.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

Mail surveys provide a relatively inexpensive approach to gathering information from a widely dispersed survey sample, and have the added benefits of eliminating interviewer bias and allowing respondents to complete the survey at their convenience. However, having the leisure of completing the survey at any time may cause some potential respondents to forget to complete the survey all-together. Furthermore, some respondents may complete the survey but not go through the extra step of sending the survey back to the researchers.

Another challenge to the response rates is the historical challenges in surveying this population. Low-income groups are hard to locate as they are more mobile and more likely to live in multifamily households. This will potentially impact the number of mail surveys that reach potential respondents. For example, pilot results yielded an overall response rate of 50% for parent-legal guardians and 70% for caregivers.

Considering all of the challenges outlined above and the pilot results, the expected overall response rate for the Parent-Legal Guardian Survey is 50% and the Caregiver Survey is 70%.

Dealing with Nonresponse

To overcome possible nonresponse, the contractor is using survey distributors – either the parent-legal guardian’s caseworker or another familiar staff member – who know the participants and have the opportunity to approach them in-person; and 2) are available to make multiple attempts at data collection. The contractor will send reminder communications as well as duplicate surveys to potential caregiver respondents at their mailing address and potentially call potential respondents who do not return the survey. The telephone reminder will include informing potential respondents of the option to administer the survey by telephone at their convenience. Respondents will also be assured as to the privacy of their information.

Additionally, the informed consent form instructs caregivers to call Westat’s toll-free 1-800 number listed on the informed consent form in the survey packet for questions and assistance completing the survey or to take the survey by telephone.

Maximizing Response Rates

PII is offering a token of appreciation to respondents to encourage them to participate in the data collection (see Section A9). Westat will distribute incentives to all participants who return a usable survey.

B4. Tests of Procedures or Methods to be Undertaken

CAPP conducted a pilot test on the study procedures in order to determine if they work as effectively as intended. The pilot was conducted as approved under the ACF/OPRE generic clearance, Pre-testing of Evaluation Surveys (0970-0355). Santa Clara County and Los Angeles-Pomona each randomly selected 5 CAPP-trained caseworkers for a total of 10 caseworkers. The sites identified and created a list of all participants that met eligibility criteria. The pilot study involved surveying 14 parent-legal guardians and 14 caregivers at each of the two sites, for a total sample of 28 parent-legal guardians and 28 caregivers. Parent-legal guardians were a convenience sample. Caregivers were a random sample from the list of eligible participants.

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The team is led by Maria Woolverton, project officer; George Gabel, project director for the PII evaluation; and Mark Testa, principal investigator for the evaluation. Additional staff consulted on statistical issues at Westat include John Rogers and Barnali Das, senior statisticians.

1 Please see section B3 (Methods to Maximize Response Rates and Deal with Nonresponse) for an explanation of our assumption of a 60% response rate

16