0990-TPP evaluation supporting_statement_B_July 2011

Outcome Evaluation of “Teenage Pregnancy Prevention: Integrating Services, Programs, and Strategies through Community-Wide Initiatives”

Part B: Statistical Methods

Heather Tevendale

Health Scientist

Phone: 770-488-6147

Email: Hrx9@cdc.gov

Fax: 770-488-6291

January 31, 2021

1. Respondent Universe and Sampling Methods

For the household administration of the CDC TPP evaluation survey (based on the already-approved PPA survey-- OMB No. 0970-0360 Exp Date 7/31/13) (Attachment C), we propose to rely on a two-stage stratified clustered sample. On the basis of preliminary estimates, we anticipate that the protocol will include an initial stage, wherein groups of contiguous census blocks within each community will be sorted into strata based on demographic characteristics. A total sample of 200 groups of census blocks (primary sampling units or PSUs) per community and administration wave will be selected from these strata with probability proportional to the PSU population. The number of PSU per strata will be proportional to the strata total population size. In the second stage, a sample of 62 households per PSU will be selected following a probabilistic systematic procedure. For that purpose, we will confirm all residences in the PSU using databases of mailing addresses available through ICF International. Eligible households from the sample of 62—i.e., households with age-eligible youth—will be identified with a brief screener (or roster). One youth from eligible households chosen randomly will be interviewed.

We anticipate the need to screen 62 households to obtain 6 respondents based on the following assumptions: a) the percentage of households with at least one member 15-19 years old is 15%; b) the average response rate is 76%; c) 15% percent of the listed housing units are vacant or not actually housing units. The proportion of households with at least one member 15-19 years old is based on the 2009 American Housing Survey. The second and third assumptions are based on the experience with the similar NGHS 2008-2010 documented in Lepkowski et al. (2010)¹.

This scheme will result in a sample of at least 1,200 youth per community and administration wave (up to 3 target and 3 comparison communities will be selected) (Table 1). To determine if the change of a proportion of interest in the intervention community is significantly different from the control community is one of the most important parameters we will estimate. In our power analysis below we focus on the change in the combined proportion of youth who (1) have not had sexual intercourse during the past 12 months and (2) have had sexual intercourse but have used contraception consistently during the past 12 months. Statistical estimates are based on the minimum expected sample of 1,200 youth even if up to1,500 youth are surveyed in communities where response rates exceed expectations.

Table 1. Expected Sample Size

Community Type	No. of sites	Average Sample	Total
Target	3	1,200	3,600
Comparison	3	1,200	3,600

2. Procedures for the Collection of Information

For each grantee community, a control community (or control unit) is selected. The purpose of a control is to provide information on what would have happened with the outcome variable of interest (e.g. teen pregnancy rates) in the target community, had this community not been exposed to the intervention.

We will compile demographic information from the census to compute a single cluster level measure to summarize numerous characteristics of clusters within the target and comparison communities (e.g., geographic sizes, population density, urban/suburban/rural classification, distribution of race/ethnicity of the population, age, educational attainment, incomes, and rates of unemployment). In addition, information about the population characteristics of teenage girls within the census tract will be enumerated, including, but not limited to, the number of girls aged 15–19 years and the percentage of girls who are African American or Latino. Baseline information on teen pregnancy and birth rates will also be compiled. Propensity score matching will be used to summarize these various community characteristics in order to select the best possible comparison control unit.

The table below presents power estimation at 5% significance level to detect the following combination of outcomes:

• The proportion of teens who had sexual intercourse during the 12 months prior to the interview decreases from 38%² to 28% in the exposed community while it remains constant in the control community

• The proportion of teens who used contraception consistently during the 12 months prior to the interview increases from 70%³ to 80% in the exposed community while it remains constant in the control community

Taken together, the percentage of teens who either did not have sexual intercourse or used contraceptive methods consistently during the 12 months prior to the interview is expected to change from 89% (100%-38% + 38%*70%) to 94% (100%-28% + 28%*80%).

This power estimation corresponds to a two-stage clustered sample of a minimum of 1,200 (up to 1,500 depending on response rates) youths (in 200 clusters) per community and wave of administration. In addition, it is assumed that clusters are matched between the exposed and the control community and reused across administrations, which introduces a correlation between clusters of about .5.

Table 2. Estimated power at 5% significance level to detect a change in the proportion of youth 15-19 who either did not have sexual intercourse or used contraceptive methods consistently during the 12 months previous to the interview

Domain	Power at 5% significance level*
Among all teenagers	91%
Among 66% of the sample	83%
Among 50% of the sample	74%
Among 33% of the sample	58%

*Power estimations are based on 1,000 stochastic simulations using a beta-binomial distribution with a correlation parameter of 0.1. Correlation between beta distributions was introduced using a Gaussian copula. The estimation of the difference in the change across administrations between the two communities was computed for each simulated sample using a logistic regression fitted by generalized estimating equation.

The proposed protocol will include a survey that will be administered in person by trained data collectors supervised by a field supervisor and a field coordinator using A-CASI technology preloaded onto tablet PCs or netbooks. Once the target and comparison communities are identified, housing will be geomapped to develop a sampling frame. Notifications will be distributed to selected households prior to the data collection visit (Attachment G). Data collectors will visit homes in person based on the sampling protocol and inquire whether the household includes youths between 15 and 19 years old and proceed to secure consent to participate from the parent and then the youth (unless the youth is aged 18 or 19). If the parent is unavailable, the data collector will follow a protocol to revisit the home at a more convenient time. Respondents with Internet accessibility will be offered flexibility in response modes; they will be provided with a URL through which they can securely access the survey while the data collector waits. This option affords the respondents a means of response that may be more familiar to them and would also include a nonaudio option.

To collect sufficient data in accordance with the sampling plan, approximately four interviews will be completed per day by each data collector. One data collector will visit eligible homes to administer the youth survey to the selected participants following approved protocols.

Field data collector training will involve 46 trainees (22 data collectors and one field supervisor from each community). The training will take place at a hotel or conference center located in close proximity to the target and comparison communities. Trainees first observe everything they will have to say or do being performed by experts (i.e., the training team), then they acquire these skills through practice, demonstrate them to each other, and finally refine each other’s performance through constructive feedback. Particular emphasis is placed on having the opportunity to become expert at all aspects of the project and to become acquainted with each of the instruments selected. If weather cooperates, trainees will get to perform their first data collection within the community before returning for a full-team debriefing. Data collectors leave training having acquired the requisite skills; in addition, they are capable of professionally representing CDC and ICF and are bonded to the project, the training team, and each other. ICF requires all staff members who work on human subject research projects, including field data collectors, to complete a training course in human subject protections annually. The training course covers the requirements specified in the codified Federal Policy for the Protection of Human Subjects.

Validation calls to scheduled respondents will be made throughout the course of data collection to ensure data collectors are following survey protocols; the protocols include arriving on time and being fully prepared to conduct the survey administration. Data collectors not meeting minimum standards will be removed from the project.

3. Methods to Maximize Response Rates and Deal with No response

We plan to ensure high response rates by distributing flyers, canvassing, and raising public awareness of the survey. Culturally tailored flyers describing the evaluation substantially increase recruitment (Yancey 2006)⁴. Additionally, the local grantee will be engaged to raise public awareness. Data collectors will be recruited from targeted communities to help maximize response rates (Yancey 2006). Interviews will be scheduled during evening and weekend hours for convenience.

We propose to collect the core survey in-person in the target and comparison communities.

Because of the sensitive nature of the questions on the youth self-report survey, other methods of

data gathering (i.e., interactive voice response system telephone surveys) would likely yield an undesirable number of incomplete surveys. Respondents will be interviewed in person in their

homes with survey instruments using A-CASI software. A-CASI technology allows us to maintain the confidentiality of responses while enhancing response and completion rates.

4. Tests of Procedures or Methods to be Undertaken

The proposed CDC TPP evaluation survey is based on a previously approved PPA evaluation survey (OMB No. 0970-0360). The survey has been modified for this data collection activity. Items that have been added to the survey solely for the outcome evaluation of “Teenage Pregnancy Prevention: Integrating Services, Programs, and Strategies through Community-Wide Initiatives” are highlighted in yellow in the attached instrument (attachment C) and are listed below in Table 3.

Table 3. New Questions added to the PPA survey for this data collection

Question No.	Question
1.6	Have you graduated from High School
1.6.2	Are you attending or have attended college
1.6.3	Are you employed at a paying job
2.1.1	Did you learn about that type of information in a class or program that occurred over multiple days
2.1.2	What was the name of the program or programs
2.2	In the last 12 months have you been exposed to media about pregnancy prevention
2.3	In the last 12 months, have you discussed pregnancy prevention or services with
2.4	If you were concerned about pregnancy prevention is there a place in the community that could provide information or other assistance
2.4.1	Given what you know about this place, how likely would it be that you would go there to access information or other assistance
2.4.2	How much would the following issues affect your decision to get information or other assistance about pregnancy prevention: cost; confidentiality; transportation
2.5 & 7.7	Have you ever received any of these services from a medical provider
2.5.1 & 7.8	Where did you receive your services
2.5.2 & 7.9	Where did you hear about the service
2.6; 7.10	In the past 12 months, have you received any of these services from a medical provider or clinic
2.6.1; 7.11	Where did you receive your services
2.6.2; 7.12	Where did you hear about the service
3.26.1	About how old was your biological mother when she had her first child
3.26.2	About how old was your biological father when he had his first child
5.10.1	During the last 30 days, on how many days did you use an illegal drug other than marijuana
6.5.1	In general, how much pressure, if any, do you feel from your boyfriend or girlfriend to have sexual intercourse?
7.17	Now please think about the last 12 months. In the last 12 months have you had sexual intercourse
7.18	How many different people have you had sexual intercourse with in the last 12 months
7.19	How many times have you had sexual intercourse in the last 12 months
7.20	When you have had sexual intercourse in the last 12 months, how often have you used a condom
7.21	When you have had sexual intercourse in the last 12 months, how often have you used any of these methods of birth control
7.22	In the last 12 months, which of the following methods of birth control have you or your partner used
7.23	When you had sexual intercourse in the last 12 months, how often did you use a condom at the same time use used [another form of birth control indicated in the response to the previous question]
7.24	In the last 12 months, did you use emergency contraception, also known as “Plan B” or Preven” or the “morning after pill
7.25	Now please think about the last 3 months. In the last 3 months have you had sexual intercourse
7.26	How many different people have you had sexual intercourse with in the last 3 months, even if only one time
7.30	In the last 3 months, which of the following methods of birth control have you or your partner used
7.31	Now, please think about the last time you had sexual intercourse. Did you use birth control
7.32	Please describe the type of birth control used the last time you had sexual intercourse
7.33	Now please think about the last time you had sexual intercourse. Were you drunk or high
7.47	To the best of your knowledge, are you pregnant now
7.48	Do you think you are probably pregnant or not

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Christine Walrath PhD

ICF International

646-695-8154

cwalrath@icfi.com

Robert Stephens PhD

ICF International

678-389-8923

rstephens@icfi.com

Michael Rodi PhD

ICF International

404-592-2116

mrodi@icfi.com

References

Abma, J.C., Martinez G.M., and Copen, C.E. (2010). Teenagers in the United States: Sexual activity, contraceptive use, and childbearing, National Survey of Family Growth 2006-2008. National Center for Health Statistics. Vital Health Statistics, 23(30).

Oakley, D. and Tsao, H.-S. (2006), A New Way of Revitalizing Distressed Urban Communities? Assessing the Impact of the Federal Empowerment Zone Program. Journal of Urban Affairs, 28, 443–471.

Rosenbaum, P. R. and Rubin, D.B. (1983). The Central Role of Propensity Scores in Observational Studies for Causal Effects. Biometrika 70, 41-55.

Rubin, D. (1997). Estimating Causal Effects from Large Data Set Using Propensity Scores. Annals of Internal Medicine, 127, 757-763

1 Lepkowski JM, Mosher WD, Davis KE, Groves RM, Van Hoewyk J. The 2006–2010 National Survey of Family Growth: Sample design and analysis of a continuous survey. National Center for Health Statistics. Vital Health Stat 2(150). 2010.

2 From the 2006-8 NSFG, 38% of youth aged 15-19 report having sexual intercourse during the past 12 months (Tables 3 & 4, Abma et al. 2010).

3 A direct estimate of teens using contraception consistently during the past 12 months is not available. We estimate this to be 70% based on the 2006-8NSFG estimate of 61% of teens using condoms consistently during the past month, condom representing 75% of the contraceptive methods used at most recent intercourse, and an a anticipated decreased of about 10% in consistency as we moved from a month to 12 months reference period (Tables 15 & 16, Abma et al. 2010).

4 Yancey, AK; Ortega, A; Kumanyika, S. (2006). Effective Recruitment and Retention of Minority Research Participants. Annual Review Public Health, 27:1-28.