A Pilot Test Summary

Attachment A

Pilot Test Summary

Purposes

The pilot test of the Family-Provider Relationship Quality (FPRQ) study was conducted to serve two main purposes. First, it has provided the data necessary to conduct item analysis and other statistical review of responses to ensure that (1) items have good distribution across the response categories and are not systematically skipped; (2) confirm that sets of items cluster as expected based on the conceptual model; and (3) conduct preliminary comparisons between provider and parent responses and among important subgroups. These analyses are necessary to ensure that each item in the measure is strong and that the overall measure reflects the conceptual model. The results of the analyses have been used to make improvements to item stem and response category wordings prior to the conduct of a larger field test, which is planned to be conducted in early 2014. Second, the pilot test provided an opportunity to test the sampling, recruitment, and training procedures and data collection systems for the field test.

Sample

The pilot test used samples of convenience. The samples were not drawn from formal sampling frames and therefore are not nationally representative of early care and education providers or of parents with young children in early childhood education programs. However, as required by the project contract, every effort was made to obtain diversity with respect to type of program (center-based, Head Start/Early Head Start, and home-based), home language (English and Spanish), race and ethnicity of parents, family income (high/low), and urbanicity (rural, suburban, urban).

We consulted with OPRE and the Office of Head Start to identify several candidate sites. Because the study’s focus groups were conducted in the Washington, DC metro area and Chicago, IL, these cities were not considered as sampling sites, to try to minimize bias due to the use of specific sites in the development process. Four cities (Seattle, WA; San Francisco/San Mateo, CA; Atlanta, GA; and Minneapolis/St. Paul, MN) were identified.  We then polled the project’s Technical Work Group (TWG) to select two sites for the pilot test; Atlanta, GA and Seattle, WA were selected for the pilot test. Center-based, Head Start, and home-based child care programs were all represented in the sample, and these programs also varied by urbanicity. The participating parents represented a range of ethnicities and home languages, as well as differing levels of economic status.

Data Collection Procedures

The pilot test of the FRPQ was conducted from February through May of 2013. During this period, our recruiters contacted child care centers, Head Start programs, and home-based providers identified from the list of programs that Westat staff compiled for each city from information available online. Programs from these lists were contacted via phone to ascertain their interest in participation in the study. Once the program directors agreed to participate in the study over the phone, we asked their permission to recruit parents and providers from their program. If permission was given, the recruiter would set a date with the director to visit the program.

During this visit, the director and providers from the program were screened for eligibility¹, and if eligible, were then given paper surveys to complete. In addition, study flyers and information were left with the program director to place in an area visible to parents to inform parents about the study and how to contact us to both participate and/or obtain more information. The flyer asked parents to call the recruiter at the listed number to complete an eligibility screener and were then told a mailed survey would be sent if eligible. Because few parents called to participate during the early stage of the data collection, subsequent program visits were often scheduled to give the recruiter an opportunity to meet and speak with parents in person to request their participation.

Four types of paper surveys were used to collect data -- director, provider, parent about provider, and parent about Head Start Family Service Workers. The results of the pilot test indicate that the data collection procedures for both the director and provider surveys worked well, and only a few minor revisions are needed for the field test. However, recruiting parents for the pilot test was challenging. We are planning to streamline recruitment procedures for parents in the field test by eliminating the parent screener and directly distributing surveys to parents at the programs during drop-off and pick-up times.

Psychometric Analysis

The psychometric analysis of the pilot data generally confirmed the conceptual model. In large part, items measured underlying constructs as intended so that there was a reasonable level of reliability, despite the fact that the sample sizes were small for psychometric analyses of new measures. With the larger samples of the field test, the conceptual model can be assessed with greater detail and certainty.

1. Results for the provider survey

The provider survey asks respondents questions about how they work with parents of children in their care. Table 1 gives the reliability analysis for the 10 scales measured in the provider survey. The second column lists the number of items in the scale, and the third column gives the reliability estimate. The numbers in red are those which fall below the rule-of-thumb criterion of .70 for reliability. Some scales were measured by few items, which could explain low reliabilities (e.g., Openness to Change and Instrumental Support).

Table 1: Reliabilities for the provider survey

Scale	Number of Items	Internal Consistency Reliability
Respect	3	0.77
Commitment and Caring	9	0.65
Openness to Change	2	0.56
Theoretical Knowledge	9	0.76
Knowledge Gathering	6	0.83
Family-Specific Knowledge	10	0.85
Relational Skills	10	0.65
Instrumental Support	3	0.47
Joint Goal Setting	7	0.85
Empower Families	3	0.76

2. Results for the parent survey about providers

The parent survey asks respondents questions about how they work with their child’s care provider. Table 2 gives the results for the 9 scales measured in the parent survey about providers. Again, some scales had reliabilities below .70, lower than the rule of thumb for adequacy. However, in each of these cases, the scales were measured by only 2 or 3 items, which could account for the poor performance. With the larger sample size of the field test, the performance of the scales can be more definitively determined.

Table 2: Reliabilities for the parent survey about providers

Scale	Number of Items	Internal Consistency Reliability
Respect	4	0.98
Commitment and Caring	14	0.87
Theoretical Knowledge	2	0.69
Knowledge Gathering	3	0.71
Family-Specific Knowledge	14	0.95
Relational Skills	10	0.78
Instrumental Practices	4	0.66
Joint Goal Sharing	8	0.85
Empower Families	2	0.42

3. Results for the director survey

The director survey asks respondents general questions about the education and child care environment, as well as program policies. Of the 53 items, four items had no variation (everyone said yes), two items had near zero correlation with the total score, and six had negative correlations with the total score. Table 3 shows the 53 items in the director survey with an “X” mark noting items (12 items) that were excluded from reliability tests because of the reasons stated above. As a result, a total of 41 items were used in the director survey analysis, resulting in a very high internal consistency reliability of .94.

Table 3: Items in director survey

Variable Name	Label	Values	Item Excluded from Analysis
Which of the following methods are used to communicate with families?
DIRQ7A	Q7a Comm W/ Family: Website	Yes/No
DIRQ7B	Q7b Comm W/ Family: Newsletter	Yes/No
DIRQ7C	Q7c Comm W/ Family: Calendar	Yes/No
DIRQ7D	Q7d Comm W/ Family: Bulletin Boards	Yes/No
DIRQ7E	Q7e Comm W/ Family: Email	Yes/No
DIRQ7F	Q7f Comm W/ Family: Text Message*	Yes/No	X
DIRQ7G	Q7g Comm W/ Family: Telephone**	Yes/No	X
DIRQ7H	Q7h Comm W/ Family: Pt Conference	Yes/No
DIRQ7I	Q7i Comm W/ Family: Talk In-Person**	Yes/No	X
Since September, has your program given any family information about the following?
DIRQ8A	Q8A Give Family Info On Employment	Yes/No
DIRQ8B	Q8b Give Family Info On Food Pantries	Yes/No
DIRQ8C	Q8c Give Fam Info On Cc Subsidies	Yes/No
DIRQ8D	Q8d Give Family Info On Tanf	Yes/No
DIRQ8E	Q8e Give Family Info On Adult Educ	Yes/No
DIRQ8F	Q8f Give Family Info On Housing Asst	Yes/No
DIRQ8G	Q8g Give Family Info On Energy Asst	Yes/No
DIRQ8H	Q8h Give Fam Info On Imm/Legal Svces	Yes/No
DIRQ8I	Q8i Give Family Info On Dom Violence	Yes/No
DIRQ8J	Q8j Give Fam Info On Substance Abuse	Yes/No
Since September, has your program provided referrals for the following services?
DIRQ9A	Q9a Health Screening Referral	Yes/No
DIRQ9B	Q9b Development Assessment Referral	Yes/No
DIRQ9C	Q9c Child Counseling Referral	Yes/No
DIRQ9D	Q9d Parent Counseling Referral	Yes/No
DIRQ9E	Q9e Social Services Referral	Yes/No
Since September, has your program offered the following to any family:
DIRQ10A	Q10a Offered Sick Care	Yes/No
DIRQ10B	Q10b Offered Extended Hours***	Yes/No	X
DIRQ10C	Q10c Offered Flexible Hours*	Yes/No	X
DIRQ10D	Q10d Offered Flexible Payment	Yes/No
DIRQ10E	Q10e Offered Transportation Help	Yes/No

Table 3: (continued)

Since September, has your program received funding from any of the following?
DIRQ11A		Q11a Code All Fund: State Pre-K*		Yes/No		X
DIRQ11B		Q11b Code All Fund: Head Start		Yes/No
DIRQ11C		Q11c Code All Fund: Child Care Development Fund*		Yes/No		X
DIRQ11D		Q11d Code All Fund: Title 1*		Yes/No		X
DIRQ11E		Q11e Code All Fund: Local Orgs		Yes/No
DIRQ11F		Q11f Code All Fund: Other*		Yes/No		X
DIRQ12&13		How Often Do You Use Parent's Feedback About Program		Never to Very Often: 4 Values
DIRQ14		Q14 Parents Can Visit Setting**		Yes/No		X
There are a variety of opportunities for parent involvement, including:
DIRQ15A		Q15a Parent Volunteering		Yes/No
DIRQ15B		Q15b Parent Providing Supplies		Yes/No
DIRQ15C		Q15c Parent Committees		Yes/No
DIRQ15D		Q15d Parent Observation**		Yes/No		X
DIRQ16		Q16 Parents Invited To Program Planning		Yes/No
DIRQ17		Q17 Suggestion Boxes/Surveys		Yes/No
DIRQ18		Q18 Activities For Fathers		Yes/No
DIRQ19		Q19 Written Material Provided to Families in All Languages		Yes/No
DIRQ20		Q20 Written Material Provided to Families at Appropriate Literacy Level***		Yes/No		X
DIRQ21		Q21 Opportunities For Family Events		Yes/No
DIRQ22		Q22 Opportunities Parent Get Together		Yes/No
The program provides parenting information through:
DIRQ23A		Q23a Parenting Workshop		Yes/No
DIRQ23B		Q23b Parent Bulletin Board		Yes/No
DIRQ23C		Q23c Parent Newsletter		Yes/No
DIRQ23D		Q23d Parent Resource Library		Yes/No
DIRQ23E		Q23e Parent Pamphlets		Yes/No

Conclusion

Overall, the surveys used in the pilot test performed well, both as comprehensive instruments and within subscales. As a result of these preliminary findings, only minor revisions were made to the surveys for the field test. Minor wording changes were made and a few poorly-performing items were removed. (These edits and the reasons behind them are further described in the Final Survey Recommendations document included in this package.) With the larger sample size of the field test, the psychometrics on data collected will be used to finalize the surveys, and these psychometric properties will be reported in the User’s Guide at the conclusion of the study.

1 Director eligibility was based on whether they cared for children 5 years of age and under, would be willing to let the recruiter talk to providers in the program, and gave permission to hand out brochures and flyers to parents of children in the program. Provider eligibility was based on whether they cared for children at least 15 hours per week and would allow the recruiter to leave brochures/flyers in the care setting for parents.

13