Supporting Statement B_1 22 12

Office of Management and Budget

Supporting Statement Part B

for text4baby Evaluation:

Collections of Information Employing Statistical Methods

January 27, 2012

This page has been left blank for double-sided copying.

OMB Supporting Statement Part B for text4baby Evaluation

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Safety Net Consumer Survey

The safety net consumer survey (which will be known as the Healthy Pregnancy and Parenting Survey, or HPP Survey) involves a two-step selection process: in the first step, four health Center-Controlled Networks (HCCNs) and one or more associated health centers will be selected and in the second step, a sample of pregnant women receiving prenatal care from the health centers will be identified.

To be eligible for selection, HCCNs will: (1) provide prenatal, obstetric, and pediatric care, (2) have one or more health centers that serve at least 320 prenatal patients within a four-month period, (3) have one or more health centers with at least 250 text4baby users within a 10-mile radius, (4) have at least one text4baby outreach partner within its catchment area, (5) have an EHR system that has been operational for at least six months (preferably operational for two or more years), and (6) agree to provide contact information for a sample of prenatal patients and an abstract of electronic health record (EHR) data for patients that consent to their release. From the 65 HCCNs funded by HRSA in 2010, only 6 met these eligibility criteria. Table 1 summarizes the steps used to select the HCCNs and health centers for the text4baby evaluation.

Four of the six HCCNs were selected to ensure diverse characteristics, including: (1) varied racial/ethnic distribution, with an emphasis on health centers that serve low-income African American and Hispanic populations; (2) geographic distribution across the four regions of the U.S., and (3) a concentration of Spanish language preferred clients in two of the sites (Table 1).

The selected HCCNs and associated health centers are as follows:

Region	HCCN Name	Health Center Name(s)	City, State
Recommended HCCNs/Health Centers
Northeast	Metropolitan Collaborative of Health Information Technology (METCHIT)	Morris Heights Health Center	Bronx, NY
Midwest	Alliance of Chicago Community Health Services	Erie Family Health Center (Alternate Health Centers: Alivio Medical Center, PCC Community Wellness Center)	Chicago, IL
South	Georgia Healthcare Systems	Southside Health Center and West End Health Center	Atlanta, GA
West	AltaMed Health Services Corporation	AltaMed Medical Group	Commerce (Los Angeles), CA
Alternate HCCNs/Health Centers
Northeast	New Community Health Centers, Inc.	North Hudson CACHD	West New York, NJ
South	Community Health Integrated Partnership	Parkwest Health Center, People Health Center, and Total Health Center	Baltimore, MD

The remaining two eligible HCCNs will serve as back-ups if an alternate choice is required. Although the final selection of the HCCNs is purposive rather than random, HRSA is using a systematic, data-driven approach to identify the HCCNs that meet our criteria and is conducting discussions with eligible HCCNs to assess their readiness and willingness to participate in the study.

The selection criteria limit the sample of HCCNs and health centers to those in larger metropolitan areas that have more active promotion and penetration of text4baby in the prenatal population. In addition, the selected health centers have larger caseloads and more sophisticated EHR systems because of their association with an HCCN. While these factors may limit generalizability of the results, they allow for an assessment of the take-up and use of text4baby by focusing on areas where text4baby subscription is more prevalent and where outcome measures can be obtained through EHRs in addition to self-reported survey data.

To achieve the target of 300 consented prenatal patients per HCCN, one or more associated health centers were selected to serve as the study sites. In three HCCNs with larger health centers, only one health center will be selected, whereas in one HCCN with smaller caseloads, two health centers will be included as study sites. Within the four selected HCCNs and five associated health centers, a sample of “current prenatal patients” will be enrolled into the study. A “current prenatal patient” would be anyone who has a prenatal care visit with the health center during the enrollment period and is between four and seven months pregnant at the time of the visit.

Due to HIPAA restrictions, health centers cannot release contact information for pregnant women to the study team without their consent. Thus, health centers will inform eligible women about the study using materials developed by HRSA (such as an advance letter and informed consent form) and will obtain their consent before releasing contact information to the study team. Parental consent will be required to interview women under age 18. The study team will work with health center staff to develop appropriate procedures for release of contact information for their patients who are under age 18. Procedures will be tailored to state-specific and center-specific requirements. The procedures and materials associated with obtaining informed consent will be reviewed and approved by the contractor’s IRB. We estimate that 13 percent of the prenatal patients will be teens who must go through an extra consent step. We expect that 80 percent of the adult prenatal patients will consent to the study, and that 48 percent of the teens will consent (80 percent parental consent and 60 percent teen consent). To get to the 1,200 consented patients, we expect the four HCCNs to have to approach about 1,630 prenatal patients (about 407 per HCCN).

To ensure statistical rigor in developing consent rates and analysis weights, the health center staff will track all eligible women regardless of whether they consent to participate in the study. Health center staff will be asked to provide to the study team a de-identified data file containing all eligible women with a unique identifier for each one as well as her age, due date, race/ethnicity, and preferred language, along with the consent forms for those who consent to participate.

The sample will include both text4baby subscribers and nonsubscribers; however, it will not be known whether a specific patient in the sample is currently a text4baby subscriber (or has ever subscribed) until she responds to the survey. Table 2 shows the expected precision (using the half-width of a confidence interval) for estimated percentages for the text4baby subscribers and non-subscribers and the minimum detectable differences (MDDs) for comparisons between text4baby subscribers and nonsubscribers under two different assumptions about text4baby subscription rates in the population and in the sample: 50 percent and 30 percent. The precision and MDDs within and across the four sites are presented. We can see that the MDDs under the two assumptions are within one percentage point of each other, showing that the MDDs are not very sensitive to the assumed subscription rate. Even if the subscription rate turns out to be lower than 30 percent, and the MDDs for quantitative outcomes increase beyond expected underlying differences, our findings will still be useful in a qualitative context.

Across the four HCCNs, we anticipate 960 completes in Round 1 of the safety net consumer survey (assuming an 80 percent response rate among the 1,200 consented women), and 768 in Round 2 (assuming an 80 percent retention in Round 2 among Round 1 respondents). These anticipated response rates are based on recent experiences with a similar population, in particular, the Healthy Start Participant Survey conducted by Mathematica for HRSA.

Consumer Focus Groups

The consumer focus groups will be conducted in the four communities in which the safety net consumer survey is conducted. Two focus groups will be conducted in each community, with 8 to 10 text4baby subscribers per group. Across the four communities, therefore, the focus groups will include a total of 64 to 80 current text4baby subscribers. The focus group recruitment strategy will rely on the text4baby platform to send a text message to the universe of text4baby subscribers in specified zip codes surrounding the selected health centers in the safety net consumer survey. This strategy will reach text4baby subscribers (pregnant and postpartum) regardless of whether they are patients in the selected safety net sites.

The text message will invite interested subscribers to call a toll-free number and those who are eligible will be given information about the dates and locations of the focus groups. To be eligible to participate, focus group participants will have been enrolled in text4baby for at least two months, live in one of the selected zip codes, and currently be receiving text4baby messages. A reminder text message will be sent to focus group participants one week in advance and again the day before.

Key Informant Interviews

The key informant interviews will be conducted in the four communities in which the safety net consumer survey is conducted. Five to ten interviews will be conducted in each community. The key informants will include health center staff (three to six interviews per community) and representatives of text4baby outreach partners (two to four interviews per community). The number of key informant interviews that can be scheduled within the allotted time will depend on logistics for scheduling the focus groups (which will occur during the same two-day site visit) and the amount of travel time required between interviews.

We will ask the director of each health center participating in the study to identify staff members who have regular interactions with pregnant women, new mothers, and infants, and who have at least basic knowledge of text4baby. We expect these staff members will include clinicians (such as physicians, midwives, and nurse practitioners) as well as other health care providers (such as case managers, health educators, and community health workers). In each community, we will request individual interviews with two to four clinicians, and arrange one or two small-group interviews with the other health care providers.

In addition, we will identify local outreach partners that participate in the text4baby public-private partnership. We will obtain a list of all text4baby outreach partners that signed a memorandum of understanding with Healthy Mothers Healthy Babies and identify those located within a 10-mile radius of the selected health centers. In each site, we will seek to include at least one local outreach partner that is a HRSA-funded safety-net provider (such as a Title V clinic, Healthy Start program, or a Ryan White provider) or a local WIC program. The other key informants may represent nonprofit organizations, health plans, or local businesses.

Stakeholder Interviews

Interviews will be conducted with up to 30 text4baby stakeholders. We will obtain a list of all partners that participate in the text4baby public-private partnership and will draw a sample of stakeholders from this list. Because these interviews will focus on national stakeholders, we will exclude local agencies and organizations, such as hospitals, health delivery networks, and clinics; health plans that serve fewer than five states; federally qualified health centers; businesses and pharmacies; Head Start and Early Head Start programs; and local Healthy Start programs. These partners will be considered for inclusion in the key informant interviews in the four selected communities.

Selection will involve a two-step process. In the first step, we will identify the U.S. Government partners, founding partners, and founding sponsor that have been involved since the beginning of the design and implementation of text4baby. These partners will be selected with certainty. We will conduct up to 10 interviews with this group of national partners. In the second step, we will select up to 20 outreach partners comprising health plans, telecom firms, state and local public health agencies or coalitions, national nonprofit organizations (especially those focusing on low-income populations), and the media. We expect that the selected partners will have been involved with text4baby for at least six months at the time of the interview and will reflect a broad array of organizations involved with program implementation. These will be selected purposively to ensure that we get a cross-section of key stakeholders within each category.

Table 1. Summary of Steps Involved in Site Selection for the Evaluation of text4baby

Step	Data Source	Approach	Results
1. Identify universe of health center controlled networks (HCCNs) and associated health centers (HCs)	HRSA Uniform Data System (UDS)	Link HCCNs with their affiliated HCs Identify HCs that directly provide prenatal, obstetric, and well-baby care	252 HCs associated with 65 HCCNs directly provide selected services
2. Identify HCs with at least 960 prenatal patients who delivered during the year	HRSA UDS	Link UDS data on number of prenatal patients during the past year Flag HCs that have at least 960 prenatal patients who delivered during the year (the threshold of 960 annual prenatal patients is a proxy for the targeted sample size of 320 in a four-month period)	Of these 252 HCs: 25 HCs have at least 960 prenatal care patients during the year
3. Assess penetration of text4baby in communities served by HCs associated with an HCCN	HMHB/Voxiva enrollment and outreach partner databases	Geocode health center and outreach partner locations for GIS analysis Conduct GIS analysis of the number of active, pregnant text4baby users within 1, 5, and 10 miles of the HCs Flag HCs that have at least 250 active, pregnant text4baby users within 10 miles of the health center (the threshold of 250 is a proxy for the targeted sample size of 160 pregnant text4baby users) Conduct GIS analysis of the number of outreach partners within 10 miles of the HCs Flag HCs that have at least one outreach partner within 10 miles of the health center (this is a proxy for active text4baby outreach in a community)	Of these 25 HCs: 6 HCs have at least 250 active, pregnant text4baby users within 10 miles of the health center Of these 6 HCs: All 6 have at least 1 outreach partner within 10 miles
4. Achieve distribution across four geographic regions	HRSA UDS	Classify HCs by region based on State location (Northeast, Midwest, South, West) Sort HCs by region and flag HCs in each region that meet the selection criteria	Of these 6 HCs: 2 are in the Northeast; 3 in the Midwest (all within the same HCCN); and 1 in the West No HCs in the South meet all these criteria, but 2 HCs in one network and 3 HCs in another are close to meeting the criteria
5. Select four communities with HCs that meet criteria	Discussions with HRSA and HCCNs	Narrow selection to four communities based on discussions with HRSA and HCCNs	Four communities with eligible HCs: Bronx NY (Morris Heights), Chicago IL (Erie), Atlanta GA (Southside/West End), and Commerce (L.A.) CA (Alta Med)

Table 1 (continued)

Step	Data Source	Approach	Results
6. Assess racial/ethnic diversity of selected HCs	HRSA UDS	Identify racial/ethnic distribution of prenatal patients in selected HCs	Most prenatal patients in Atlanta (Southside and West End HCs) are African American Most prenatal patients in Chicago (Erie HC) and Commerce (Alta Med HC) are Hispanic Morris Heights in the Bronx serves a mix of African American and Hispanic prenatal patients
7. Assess EHR capacity of selected HCs	HRSA EHR System Extract	Ascertain the type of EHR system used by each HC and the extent of EHR system implementation	Two HCs (Morris Heights and Erie) use GE Centricity The Atlanta HCs use Visionary Dream Alta Med uses NextGen By the time the study is conducted, EHR systems will be fully implemented in the selected HCs These are commonly used EHR systems

Table 2a. Safety Net Consumer Survey (Round 1, Pregnant Women) Sample Sizes and Precision (Half-Width of Confidence Interval Around Percentage Given in Column)*

			Per Site				Across Four Sites
Subscription Rate	Text4baby Group		Responding Sample	Precision for Estimated Percentage of			Responding Sample	Design Effect (Within-Site Clustering, =.01)	Precision for Estimated Percentage of
				50%	25%				50%	25%
50%	Subscribers		120	7.5%	6.5%		480	2.19	6.6%	5.7%
	Non-subscribers		120	7.5%	6.5%		480	2.19	6.6%	5.7%
30%	Subscribers		72	9.8%	8.5%		288	1.71	7.6%	6.6%
	Non-subscribers		168	6.4%	5.5%		672	2.67	6.2%	5.4%
-----	All Prenatal		240	5.3%	4.6%		960	3.39	5.8%	5.0%

*For site-specific estimates, we use a 90% confidence interval. For overall estimates (across all four sites), we use a 95% confidence interval.

Table 2b. Safety Net Consumer Survey (Round 1, Pregnant Women) Sample Sizes and Minimum Detectable Differences (MDDs)*

	Per Site						Across Four Sites
Subscription Rate	Responding Sample			MDD for Estimated Percentage of			Responding Sample			Design Effect (Within-Site Clustering)			MDD for Estimated Percentage of
	S	NS		50%	25%		S	NS		S	NS		50%	25%
50%	120	120		14%	12%		480	480		2.19	2.19		13%	12%
30%	72	168		15%	13%		288	672		1.71	2.67		14%	12%

*MDDs are computed for a two-sided test of the difference in percentages. For site-specific estimates, we use alpha=.10 and 70% power. For overall estimates, we use alpha=.05 and 80% power.

S = Subscribers; NS = Non-subscribers

2. Procedures for the Collection of Information

Safety Net Consumer Survey

Data collection procedures will engage HCCNs and associated health centers to help HRSA enroll eligible women into the survey on a rolling basis over a four-month period (or as long as is needed to reach the target number within each HCCN). The telephone survey will begin three weeks after enrollment starts to ensure that sufficient cases have accumulated so that cases can be worked efficiently within a 16-week timeframe. The survey will use computer-assisted telephone interviewing (CATI) software and associated management technology to interview the women. These and other procedures will minimize burden on the HCCNs/health centers and the women participants and improve data quality. Our response to question 3 on page 9 below goes into greater depth on steps used to maximize response, reduce burden to women, and promote accuracy and completeness of information collected. Specifically, the following data collection procedures are planned:

Enrolling Eligible Women. HRSA will work with HCCNs and associated health centers to customize site-specific enrollment procedures that include (1) identifying and keeping track of all eligible pregnant women, that is, women who have received prenatal care services at the health center at least one time during the enrollment period and are between 4 and 7 months pregnant at the time of the visit, (2) enrolling eligible women into the study and obtaining their signed consent to participate using materials discussed below, and (3) providing sampling information, including a list of all eligible women plus copies of signed consent forms, to the study team on a weekly basis. In signing the informed consent form, the women acknowledge that they understand (1) the survey purpose as laid out in the materials and explained by site staff, (2) any possible risks and benefits associated with participating in the survey, including possible disclosure of their identity, and (3) the voluntary nature of the survey. The informed consent protocol will also include a request for permission to access each woman’s EHR and link her data to her survey responses. At a minimum, the customized enrollment materials will consist of (a) an advance letter explaining the study, assuring the sample member of the privacy of their identity and information, and mentioning the $20 gift card incentive, (b) a brochure providing more details about what is expected and the nature of the information sample members will be asked to provide, (c) the enrollment and consent form, and (d) customized materials the HCCN and/or health center will need to keep track of the enrollment process. HRSA expects to enroll approximately 300 women per HCCN and to complete Round 1 interviews with 240 women per HCCN. In addition to training on enrollment, the health center staff will be trained to convey sample information on all identified eligible women, whether or not they decide to enroll in the survey and sign the consent form. This information will enable HRSA to establish the total eligible sample and thus calculate a response rate and analysis weights.

Implementing the Survey in Blaise and Survey Management Software. The pretested text4baby questionnaires will be programmed into CATI using Blaise software. The Spanish translation of the questionnaire will be implemented and tested in CATI as well. The instruments will undergo extensive testing by programmers, survey managers, and HRSA. Training materials will be finalized after the implementation and testing in Blaise is complete. In addition, Survey Management Software (SMS) will be customized to manage the text4baby surveys. The SMS will track the status of cases as they are called, scheduled, and interviewed and will be the source of all production reports. The SMS not only helps to improve survey productivity, but also is an important resource in maximizing response rates by scheduling calls at different times and days of the week.

Interviewer Training. The basis of interviewer training will be a training manual. The core sections of the manual will include background information on the text4baby surveys and its target population of pregnant and postpartum women receiving health services at the selected health centers, question-by-question specifications, basic respondent interaction techniques, and strategies for averting refusals. Interviewers will receive 8 hours of training for the first survey and an additional 4 hours for the second survey. The training manual will serve as a day-to-day reference for the interviewers’ use.

Collecting the Information. Data will be collected using CATI over a 16-week field period for each round of the survey. In addition to conducting the surveys in English and Spanish, interpreters in less commonly used languages will be available to ensure that language is not a barrier to participating in the survey. To encourage participation, a $20 respondent payment in the form of a gift card, tailored to the health center, will be mailed to all respondents who complete the interview. As part of the pre-interview consent process, respondents will be asked to grant permission for the HCCN/health center to provide a subset of EHR data that will later be combined with their survey data. The EHR data will be abstracted after the conclusion of the Round 2 interviews.

An important part of data collection involves monitoring the work of the interviewers. All of the interviewers’ first few interviews will be monitored and subsequently 10 percent of the interviews will be monitored. Feedback will be given to interviewers immediately about their performances.

Locating is another important element of data collection. When interviewers are unable to reach a respondent by telephone after trying on several days at different times of day, the case will be sent to locating staff who will carry out internet searches, telephone 411 Reach searches, as well as Google and White Pages searches. In addition, health center staff may be able to provide updated contact information on hard-to-locate respondents.

Debriefings. During the first two weeks of data collection, data collection staff will debrief several times with interviewers to identify and respond to any questions raised by respondents or interviewers. A proactive stance means that issues are identified before they evolve into problems.

Weekly Reports. During the field period, weekly reports will be generated to monitor actual production compared to the production forecast. Weekly reports provide the tool necessary to take corrective action when needed, for example, by identifying patterns in nonresponse among different sample members.

Data Cleaning. At the end of the survey all data will be compiled. Data will be cleaned based on specifications developed between the programming and survey staff. The final data file will be used for analysis.

Consumer Focus Groups

Two groups will be conducted in each community, with 8 to 10 text4baby subscribers per group, for a total of 64 to 80 current text4baby subscribers. The groups will be conducted in an accessible location, such as a public library or community center. We will ensure that the space is private (such as an enclosed conference room) to maintain confidentiality and minimize distractions. In each site, we plan to schedule an afternoon and evening focus group on consecutive days to accommodate different schedules. Each focus group will last a total of 90 minutes. Fifteen minutes will be devoted to intake (including obtaining consent), welcome, and introductions; 60 minutes to discussion; and 15 minutes to wrap-up and distribute the gift cards. The focus groups will be taped for transcription. Upon arriving, participants will receive a participant information form (PIF) to collect demographic information and responses to close-ended questions about their experience with text4baby. After completing the focus group, women will receive a $20 gift card for their participation. Each focus group will be staffed with a moderator and facilitator. The facilitator will be responsible for intake, processing gift cards, welcoming late arrivals, recording the discussion, and taking notes. The moderator will lead the group discussion, ensuring that all participants have an opportunity to speak, drawing out those who are reticent, and cueing participants to share the diversity and similarity of their experiences.

Key Informant Interviews

All key informant interviews will be in person. Individual interviews with clinicians will last 30 to 45 minutes. Group interviews with other professionals will last 45 to 60 minutes. At each site, we will attempt to schedule individual and group interviews to take place over two days and within regular work hours. The two-person interview team will include a senior person to lead the interviews and a junior person to help schedule, facilitate, and take detailed notes. We will audio-record the interviews if respondents agree. We do not plan to transcribe the recordings, but to have them available to clarify our notes or extract verbatim quotes.

Stakeholder Interviews

Selected stakeholders will be contacted by phone or email to introduce the study and schedule an interview. All interviews will be conducted by phone and will use a semi-structured discussion guide that contains a core module for all stakeholders and targeted, supplemental modules for selected stakeholders, including mobile technology partners, content development partners, health plans, and media partners. Each interview will last between 30 and 60 minutes. The two-person interview team will include a senior person to lead the interviews and a junior person to help schedule, facilitate, and take detailed notes. We will audio-record the interviews if respondents agree. We do not plan to transcribe the recordings, but to have them available to clarify our notes or extract verbatim quotes.

Information Collection Schedule

Table 3 shows the survey schedule. HCCNs and associated health centers will be recruited one time only, before Round 1 of the survey. After OMB approval is received, enrollment and consent procedures will be adapted for each HCCN/health center’s work flow and health center staff will be trained on how to implement the enrollment and consent procedures for the study. Upon completion of the training, health center staff will begin enrolling eligible pregnant women using the materials described above. Information collection from pregnant women will begin in March and continue for 16 weeks. The contractor will share weekly production reports with HRSA, and conduct data quality reviews periodically during the field period.

Attachment A includes all of the data collection instruments for the text4baby evaluation. Attachment A1 consists of the Round 1 Survey (Pregnant Women). Attachment A2 consists of the Round 2 Survey (Postpartum Women). Attachment A3 consists of the focus group protocol and participant information forms. Attachment A4 consists of the key informant interview protocol (core and targeted modules). Attachment A5 consists of the stakeholder interview protocol.

Table 3. Survey Schedule

Task	Time Schedule
Develop data collection tools	September 2011
Receive OMB approval	January 2012
Safety Net Consumer Survey
Train health center staff on enrollment process	February 2012
Enroll study sample	February 2012-May 2012
Collect survey data (Round 1)	March 2012-June 2012
Analyze data (Round 1)	July 2012-September 2012
Collect survey data (Round 2)	December 2013-March 2013
Obtain EHR data	March 2013
Analyze linked survey/EHR data (Rounds 1 & 2)	April 2013-June 2013
Consumer Focus Groups and Key Informant Interviews
Conduct site visits	April 2012-May 2012
Analyze data	June 2012-August 2012
Stakeholder Interviews
Conduct interviews	March 2012-April 2012
Analyze data	May 2012-June 2012
Reports and Presentations
Interim evaluation report	September 2012
Interim briefing	October 2012
Final synthesis report	August 2013
Final study briefing	September 2013

3. Methods to Maximize Response Rates and Deal with Nonresponse

Safety Net Consumer Survey

The data collection procedures discussed below were designed to maximize response rates and to promote the accuracy and completeness of information collected.

Obtaining Assistance from Health Centers. During the first round of data collection, we will build upon the health center staff relationships with their patients to promote the survey among eligible patients and engage them in participating in the surveys. Before data collection starts, participating HCCNs and associated health centers will be briefed and later trained to ensure they have a full understanding of the eligibility criteria, the informed consent procedures for adults and minors that comprise enrollment, and the data collection plan. HRSA will train each health center to use procedures customized to its staffing arrangement and work flow. This will ensure that all sampling and consent procedures are clear and implementable at the site level. In addition, because the health centers will be sending completed consent forms weekly to the contractor, we expect there will be a minimal gap between the enrollment date and the interview date. This has two important implications: the contact information provided on the consent form will be fresh and the enrolled participant will remember enrolling and thus be more likely to complete the telephone interview.

Importantly, during the first round of data collection, the health centers will report weekly about the enrollment process: how many eligible women have been identified and how many have enrolled (signed a consent form). As the enrollment process will be ongoing for approximately 12 weeks, the weekly report is critical to monitoring the progress of the sample accrual. This, in turn, is important to generating a response rate report for the survey. The report will enable HRSA to understand how well the completion projections are being met and when enrollment is complete.

During Round 2, sample members for the postpartum survey will be women who responded to the Round 1 survey pregnant women’s survey. Health centers will not need to assist with enrollment. However, the health centers will be asked to provide updated locating information for women who are unlocatable and also encouragement to reluctant participants.

Implementing the Instruments in Blaise Survey Software. Implementing the survey in Blaise software will provide a controlled way to collect data that ensures high quality and consistency by and enforcement of rules to avoid various kinds of error. The Blaise program will (1) control the routing through the questionnaire, thus avoiding pathing errors; (2) control response ranges so that out of range values are checked and updated in real time by the interviewer; and (3) make consistency checks to ensure that the respondent’s answers are consistent throughout the questionnaire. In addition, it will fill responses from previous asked questions, thus helping interviewers smoothly administer the survey. Finally, Blaise software allows the review of metadata for management purposes.

Conducting Interviewer Training. Well-trained interviewers are able to help improve response rates by successfully convincing participants to take part, avoiding refusals in reluctant or hostile respondents, and by conducting the survey in a professional but friendly manner that keeps the respondent actively engaged in the interview. Text4baby training will be based on a detailed training manual and supplemented by practice exercises in gaining cooperation, and multiple practice exercises to help them become comfortable with the various major paths of the surveys. During interviewer training, problems with language or routing are sometimes identified. HRSA will leave sufficient time between training and the start of interviewing to correct and test any errors discovered during training.

Locating Respondents. If a text4baby sample member is unlocatable, she cannot be interviewed; thus successful locating becomes a critical step in achieving a high response rate. When interviewers are unable to reach a sample member by telephone, they send the case to highly trained locating staff who have skills using internet searches, telephone 411 Reach searches, as well as Google and White Pages searches.

Monitoring Interviews to Ensure High Quality Data. Monitoring how interviewers conduct surveys, especially the early ones, is critical to ensuring the high quality of the data. Highly trained monitors will review in “real time” interviews conducted by interviewers. During the first week of interviewing, they will monitor the first case completed by each interviewer. After this point, they monitor 10 percent of all interviews. The monitor listens on the telephone while sitting in front of two computer screens: one screen shows the keystrokes the interviewer is using to record answers, the other screen provides an assessment form where the monitor records the quality of the interview. At any point the interviewer makes a serious error, potentially leading to missing information, the monitor will approach the interviewer and provide immediate corrective feedback.

Debriefing Interviewers to Identify Problems Early. HRSA regards debriefing interviewers as a critical step in quality control of the surveys. Survey staff will be debriefed several times in the first week or two of data collection to identify problems in the question language or survey routing. Corrections will be made to errors identified during the debriefings. Update sheets will be sent to all interviewing staff to apprise them of updates to procedures, thus ensuring consistency across all staff.

Reviewing Data Frequencies. Frequency reviews are an important tool in ensuring data quality. To determine whether the instrument is performing as specified, survey frequencies will be reviewed after the first 50 cases are completed. If any programming errors are detected, for example, erroneous skip logic or inadequate range specifications, interviewing will stop until the error has been corrected and the correction thoroughly tested. If any missing data need to be retrieved from respondents, specially trained quality control staff will telephone and obtain the missing information. Frequencies will be run a second time to verify that the error has been corrected. We will also ask for a preliminary file of all eligible women for whom consent was attempted to ensure that the final file will be usable to construct analysis weights and compute response rates.

Minimizing Nonresponse Bias. Nonresponse bias is a function of both response rate and how different respondents and nonrespondents are with respect to a particular measure. We will do everything possible to maximize the response rate, which will help mitigate the risk of nonresponse bias. However, because it is unlikely that we will achieve an 80 percent response rate or higher, and because we cannot directly determine how different respondents and nonrespondents are on our key outcome measures, we plan to compare what is known about both respondents and nonrespondents to get a sense of the risk for nonresponse bias. This in turn will point us to the best set of characteristics to use when adjusting the sampling weights for nonresponse. We will explore whatever data are available for both respondents and nonrespondents, using modeling to determine which ones are significantly related to the propensity to respond, and which of these are likely to be related to our key outcomes. We will then construct a propensity score from this logistic regression model to use to adjust our sampling weights for nonresponse.

Consumer Focus Groups, Key Informant Interviews, Stakeholder Interviews

Response to the three qualitative components—consumer focus groups, key informant interviews, and stakeholder interviews—is expected to be high because of the salience of text4baby as an innovative mobile health technology. Outreach to prospective focus group participants will take place via a text message on their cell phones, increasing the likelihood of reaching text4baby subscribers on a timely basis. They will be offered a respondent payment of $20 to compensate them for their time. In addition, participants will be given a choice of two times (one afternoon and one evening) and the groups will be held in convenient locations to increase response. Reminders will be sent to confirmed participants by text message one week in advance and again the day before.

Similarly, the key informant interviews and stakeholder interviews will be scheduled at the convenience of the respondent and will last between 30 and 45 minutes. A response rate of 95 percent is expected for the key informant and stakeholder interviews based on previous experience with similar activities and a typically high level of motivation from the partners involved.

4. Tests of Procedures or Methods to be Undertaken

HRSA carried out a pretest of the safety net consumer survey, including both the pregnant women’s survey and the postpartum women’s survey. Key findings were that the timing of the survey was in line with the budgeted survey length (20 minutes), that the survey worked well but needed a few wording changes (particularly for the usual source of care and health insurance questions), and that two questions needed to be moved to the beginning of the surveys in order to make the survey entry flow more smoothly. Attachment B contains the Pretest Report and Recommendations.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

Members of the text4baby TAG were consulted on a monthly basis about the substantive and methodological aspects of the study. Their recommendations were incorporated into the study design and instruments on an ongoing basis. See Attachment C for a list of TAG members.

The person responsible for receiving and approving contract deliverables is Emily DeCoster, HRSA.

The design, data collection, and analysis are being conducted by Mathematica Policy Research, prime contractor, and Public Health Institute, subcontractor.