Supporting Statement B
Title X
Implementation Study
Submitted to
Office of Management and Budget
Office of
Information and Regulatory Affairs
Submitted by
U.S. Department of Health and Human Services
Office of the Assistant Secretary for Health
Office of
Population Affairs
Point of contact: Roshni Menon
Contents
B1. Respondent Universe and Sampling Methods 3
B2. Procedures for Collection of Information 4
B3. Methods to Maximize Response Rates and Deal with Non-Response 6
B4. Test of Procedures or Methods To Be Undertaken 7
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 7
Attachments
ATTACHMENT A: title x authorizing legislation
ATTACHMENT B: Research question and data source crosswalk
ATTACHMENT C: grantee web survey advance email
ATTACHMENT D: GRANTEE WEB SURVEY Invitation email
ATTACHMENT E: grantee web survey reminder email
ATTACHMENT F: Grantee phone interview invitation email
ATTACHMENT G: LISTENING VISIT INVITATION EMAIL
ATTACHMENT H: Subject matter expert interview invitation email
Attachment I: 60 day federal register notice
Instruments
INSTRUMENT 1: Grantee web survey
INSTRUMENT 2: Grantee interview topic guide
INSTRUMENT 3: listening visit topic guide for clinic administrators
INSTRUMENT 4: listening visit topic guide for clinical service providers
INSTRUMENT 5: listening visit topic guide for community Partner and outreach staff
INSTRUMENT 6: client web survey
INSTRUMENT 7: Subject Matter Expert Topic Guide
This information collection request (ICR) includes the following activities associated with the Title X Implementation Study: grantee web survey and telephone interviews; listening visits with grantee sub-recipients and service delivery site staff and partners (in person or virtual); and interviews with Title X subject matter experts.
Below we describe the respondent universe and sampling methods for each instrument.
Grantee web surveys and telephone interviews (Instruments 1 and 2). The respondent universe for these instruments is a census of all 89 Title X grantees funded in 2022. A census is appropriate because the population is small, each grantee is unique, and a sample of the grantees might miss some key practices. Each grantee’s project director will be the respondent for the web survey. For the telephone interviews, we will ask grantees to select up to two staff who are knowledgeable about the grant activities (such as the project director, medical director, nurse lead, or other medical professional). To facilitate administration of these instruments, the Office of Population Affairs (OPA) will supply the study contractor, Mathematica, with contact information for the grantee project directors.
Listening visits. After administering the grantee web survey and telephone interviews, the study team will select a purposive sample of up to 40 sub-recipients and/or service delivery sites (referred to as sites) to participate in listening visits. These sites are the physical locations (health clinics, hospital clinics, and so on) through which clients receive Title X services. To answer the study’s research questions and document the implementation of Title X services in various settings, the study team will select sites purposively accounting for the following criteria:
Size and location: geographic representation across the United States; range in number of clients served annually (inclusive of sites serving fewer clients and those serving many clients)
Type of grantee: (1) state-run agency, (2) large nonprofit, or (3) other/mix
Population served (residents of urban and rural areas)
Role in Title X network, such as service site only, sub-recipient and service site, or grantee and service site
Clinic type, such as federally qualified health center, health department, nonprofit clinic, or hospital clinic
Client population, for example, racial/ethnic diversity and age
The study data collection instruments require responses from several people at each site.
Clinic administrators (Instrument 3). For each site, the study team will interview up to two clinic administrators who handle day-to-day clinic operations. Because of the different structures of service delivery networks across Title X grantees, these respondents could have various job titles, such as clinic manager, nurse lead, or medical director.
Clinical service providers (Instrument 4). For each site, the study team will interview an average of four clinical service providers. The job titles of clinical service providers can vary greatly based on clinic structure. Possible respondents include doctors, physician assistants, nurse practitioners, certified nurse midwives, registered nurses, public health nurses, social workers, licensed practical nurses, health educations, and clinic aids.
Community partners and outreach staff (Instrument 5). For each site, the study team will interview up to two community outreach or partner staff. The specific job duties of these staff vary across sites but can include developing and carrying out strategies for building community buy-in and trust, overseeing community engagement and outreach strategies, and forming or maintaining community partnerships.
Client web survey (Instrument 6). For up to 10 of the sites selected for listening visits, the study team will conduct an intercept survey of clients older than 18 who speak English or Spanish and whose visits are funded by Title X. The sample of sites selected for this survey will be a nonprobability-based purposeful sample as opposed to probability based. The study team will select sites based on conversations with clinic administrators about the feasibility of conducting an intercept survey in their locations. In each participating site, the survey will be administered to up to 30 clients over a set four-week period.
Title X subject matter experts (Instrument 7). The study team will interview up to 25 Title X subject matter experts. The study team will identify candidates through discussions with OPA and by reviewing research studies related to Title X.
Grantee web surveys. Before the survey, the study team will send each Title X project director a notification email that briefly describes the study and the grantee survey (Attachment C). The email will include a description of the topics covered in the survey and will provide the study team’s contact information; in case the respondent has any questions. The email will encourage participation by highlighting the purpose and intended contributions of the study. The email will also describe the time needed to complete the survey and provide the deadline for completing it. The study team will follow up by email with step-by-step instructions for accessing and completing the survey (Attachment D). The team will field the survey for four to six weeks. Nonrespondents will receive periodic reminders to complete the survey (Attachment E), and the study team will inform OPA about the response rates so federal project officers can follow up with nonrespondents. The survey will take about one hour to complete.
The study team will field the survey on the web to enable grantees to respond to the survey on their own time using their preferred electronic devices, such as a smartphone, tablet, laptop, or desktop computer. Respondents can pause, with their responses saved, and restart the survey using a different device. They can also go back and change responses, if needed, before final submission. The study team will program the survey with checks to prevent missing, inconsistent, or implausible responses.
Grantee telephone interviews. The study team will conduct telephone interviews on a rolling basis, about two to three weeks after a grantee submits its web survey. The study team will email the Title X project director to invite up to two staff who are knowledgeable about the grant activities to participate in a group interview (Attachment F). The invitation will describe the purpose of the interview and suggest some potential dates and times for the call. The study team will then follow up by telephone to schedule the interview at a time that is convenient to the respondents. The team will send a reminder email a few days before the interview. Up to two researchers from the study team will conduct each interview. Before the interview, the interviewers will ask for each respondent’s verbal consent to participate in the interview and for interviews to be recorded.
Listening visits. For the sites selected for listening visits, the study team will obtain contact information for a clinic administrator from OPA or a Title X grantee project director. The study team will then contact the clinic administrator to schedule the session and identify appropriate respondents for interviews (Attachment G). The team will also discuss with the clinic administrator if the interviews should be completed in person or virtually by telephone or a web-based platform. The decision about the mode of listening visit will consider the proposed timing of the visit, coordination with other program site visits, and minimizing burden on clinic staff. To simplify scheduling, we will hold small-group interviews for clinical service providers, community partners, and outreach staff from the same organization. For sites with only a single staff member in these positions, the study team will interview that person individually. Up to two researchers from the study team will conduct each interview. Before the interview, the interviewers will ask for each respondent’s verbal consent to participate in the interview and for interviews to be recorded.
Client surveys. For sites where it is feasible to conduct a client intercept survey (we estimate this will be up to 10 sites), the study team will offer a short survey at the end of scheduled appointments to English- and Spanish-speaking Title X clients who are older than 18. During a set, approximately, four-week data collection period, members of the site’s clinical or administrative staff will distribute survey flyers to all Title X clients. The flyers will have individualized links to the web survey (that recipients can only use once). The study team will train site staff in data collection procedures, including the importance of distributing the flyer to all eligible clients served during the data collection period, and explaining the study and survey in a consistent way. Site staff will have study team contact information they can call if needed during the data collection period. The initial training will be followed by an additional call with all participating staff after the start of data collection to confirm procedures and provide any needed technical assistance. Clients will have the ability to complete the survey on their own time, either while at the site or after the visit. The initial log-in page of the survey will contain consent language, informing respondents that their participation is voluntary and describing planned uses of the data.
At each site, site staff will administer the survey for about four weeks or until 30 clients have completed the survey. If the number of responses at the end of the four-week period is fewer than 30, the study team will extend the administration period and engage with site staff to ensure that information about the survey is being shared with all eligible clients. The administration period will end any time a site reaches the targeted sample size of 30 clients.
Subject matter expert interviews. The study team will identify candidates through discussions with OPA and by reviewing research studies related to Title X. The study team will email each selected candidate to invite them to participate in an interview (Attachment H). The invitation will describe the purpose of the interview and suggest some potential dates and times for the call. The team will then follow up by telephone to schedule the interview at a time that is convenient to the respondent. The team will send a reminder email a few days before the interview. Before each interview, the interviewer will ask for each respondent’s verbal consent to participate in the interview and for interviews to be recorded.
Grantee web surveys and telephone interviews. OPA expects a high response rate (greater than 90 percent) for the grantee web survey and telephone interviews, for several reasons. First, grantee project directors who will respond to the survey are heavily invested in providing family planning services and, therefore, OPA expects they will be motivated to participate. Second, the study team will use several strategies to contact nonrespondents and encourage their participation (see the next paragraph on maximizing response rates). The expected response rate is adequate for answering the study research questions and describing the Title X grantees.
The study team will use several strategies to achieve the intended response rates. OPA will broadcast information about the study to all grantees through grantee webinars and regular email communication and will include a discussion on why participation matters for each grantee and the larger field. In addition, the federal project officers assigned to each grantee, along with the study team, will provide information about the study and send reminder emails (Attachment E) after approval of this ICR. The study team will follow up with nonrespondents to encourage their participation by highlighting the importance of the study. Federal project officers might also contact nonrespondents to encourage their participation. If the survey response rate is less than 90 percent, the study team will conduct a nonresponse analysis to identify any systematic differences between respondents and nonrespondents.
Listening visits. OPA expects that 80 percent or more of the selected staff will choose to participate in the interviews. This expectation is based on experience with similar instruments in studies with similar populations. The study team will use the following strategies to maximize the response rates:
Establish trust with all interviewees. To establish trust, the study team will clearly communicate to all study participants the purpose of the data collection and use of eventual findings. The team will also ask the grantee project directors to discuss the study with their participating sites.
Offer flexibility in mode. The study team will coordinate with an administrator at each site to determine the mode, either in person or virtual, that will work best for their staff.
Offer flexibility in timing. The study team will coordinate with staff at sites to find the best time to conduct interviews, including during lunch hours or after clinic hours.
Send advance and reminder emails for interviews. The study team will send advance emails to potential interviewees requesting their participation, and will send a reminder email a few days before the scheduled interview.
Design interview topic guides in a manner that minimizes burden. The study team will use expert review of all interview topic guides to ensure interviewers only ask the most relevant questions.
Client web survey. The response rate for the intercept survey of clients is expected to be around 70 percent. The data collection methodology proposed is similar to waiting room and patient satisfaction surveys, which often see high response rates.1,2 In a review of response rates in published patient satisfaction surveys, studies with face-to-face recruitments had an average response rate of 76.9 percent.3 However, in the absence of study team members on site, OPA anticipates the response rate will be lower, closer to 70 percent. The study team will use several strategies to maximize our response rates. The client survey is designed to take respondents only about 10 minutes to complete at the end of their appointments. They can also take the link to the survey with them if it easier for them to complete it on their own time. The study team will train staff to approach all clients and encourage participation in the survey. To increase the response rate, the team also intends to provide a token of appreciation to thank participants for their time and effort in the study (please see Section A.9 for more information about the token of appreciation).
To assess response rates and non-response, each site will provide the study team with the total number of Title X clients served during the data collection period and will aggregate demographic information for these clients. The study team will use this information to calculate the response rates for each site and conduct a non-response bias analysis based on client demographics.
Grantee web survey. We developed items from the grantee web survey based on a review of the following: 2019 Title X grantee applications, the 2022 Notice of Funding Opportunity for Title X, relevant materials from the Reproductive Health National Training Center, a survey of clinic administrators at reproductive health centers used in the evaluation of The Right Time Initiative,4 and the 2010 Survey of Clinics Providing Contraceptive Services, funded by Guttmacher Institute.5 Staff from up to three Title X grantees will review the survey to ensure items are understandable and use language and terms familiar to grantee staff.
Client web survey. We selected items for the client survey based on a review of reproductive health client surveys, updated to align with the research questions for this study. The survey includes the Person-Centered Contraceptive Counseling measure, endorsed by the National Quality Forum in November 2020. Items on health literacy were based on previously validated health literacy screening items.6 Demographic items were based on items used in other federal surveys approved by the Office of Management and Budget (OMB). The gender identity item is based on the National Academies of Sciences, Engineering, and Medicine recommendations for measuring sex, gender identity, and sexual orientation7. Race and ethnicity items were based on OPA’s IMAGIN Frontline Staff Survey (OMB Control #0990-0469).
The study contractor (Mathematica) pretested the survey with five women, recruited from Title X health centers or organizations serving a similar low-income population, to ensure that questions were understandable, use language and terms familiar to respondents, and were consistent with the concepts they aim to measure; to identify typical instrumentation problems, such as question wording and incomplete or inappropriate response categories; to measure the response burden; and to confirm that there were no unforeseen difficulties in administering the instrument. The contractor administered the pre-test using a hard copy, paper version of the survey, and then held a short debriefing session with Mathematica staff. Three pretests were completed in English and two in Spanish. The contractor made changes to the survey based on participants’ feedback. The final version of the survey will be programmed as a web-based survey. Mathematica staff will thoroughly test the web survey before fielding.
OPA contracted with Mathematica to develop the data collection instruments. Mathematica will lead the data collection activities described in this ICR. Table B.5 lists the people responsible for instrument design, data collection, and the statistical aspects of the data collection, and includes their affiliations, telephone numbers, and email addresses.
Table B.5. People that OPA consulted on the instrument development and analysis
Name |
Affiliation |
Telephone and email |
Roshni Menon |
Office of the Assistant Secretary for Health Office of Population Affairs, Division of Research and Evaluation |
(240) 453-6175 Roshni.Menon@hhs.gov |
Amy Margolis |
Office of the Assistant Secretary for Health Office of Population Affairs, Division of Research and Evaluation |
(240) 453-2820 amy.margolis@hhs.gov |
Amy Farb |
Office of the Assistant Secretary for Health Office of Population Affairs, Division of Research and Evaluation |
240-453-2836 Amy.farb@hhs.gov |
Shenena Merchant |
Office of the Assistant Secretary for Health Office of Population Affairs, Division of Research and Evaluation |
(240) 453-2848 Shenena.merchant@hhs.gov |
Brian Goesling |
Mathematica |
(609) 945-3355 bgoesling@mathematica-mpr.com |
Jennifer Blum |
Mathematica |
(510) 285-4627 jblum@mathematica-mpr.com |
Subuhi Asheer |
Mathematica |
(609) 936-2715 sasheer@mathematica-mpr.com |
So O’Neil |
Mathematica |
(617) 301-8975 |
Katie Adamek |
Mathematica |
(617) 583-1940 kadamek@mathematica-mpr.com |
Jennifer Walzer |
Mathematica |
(312) 994-1042 jwalzer@mathematica-mpr.com |
Maggie Magee |
Mathematica |
(510) 830-3705 mmagee@mathematica-mpr.com |
Erin Boyle |
Mathematica |
(609) 945-6608 eboyle@mathematica-mpr.com |
Sarah Wagner |
Mathematica |
(202) 250-3521 swagner@mathematica-mpr.com |
Carla Chavez |
Mathematica |
(617) 588-6735 cchazez@mathematica-mpr.com |
1 Ongena, Y.P., M. Haan, T.C. Kwee, and D. Yakar. “Shaking Hands in a Busy Waiting Room. The Effects of the Surveyor’s Introduction and People Present in the Waiting Room on the Response Rate.” Survey Methods: Insights from the Field, 2021. Available at https://surveyinsights.org/?p=14494.
2 Lenehan, J.G., K.C. Leonard, S. Nandra, C.R. Isaacs, A. Mathew, and W.A. Fisher. “Women’s Knowledge, Attitudes, and Intentions Concerning Human Papillomavirus Vaccination: Findings of a Waiting Room Survey of Obstetrics-Gynaecology Outpatients.” Journal of Obstetrics and Gynaecology Canada, vol. 30, no. 6, June 2008, pp. 489–499. doi:10.1016/S1701-2163(16)32864-X.
3 Sitzia, J., and N. Wood. “Response Rate in Patient Satisfaction Research: An Analysis of 210 Published Studies.” International Journal for Quality in Health Care, vol. 10, no. 4, August 1998, pp. 311–317. https://doi.org/10.1093/intqhc/10.4.311.
4 Hoe, Emily, and So O’Neil. “Lessons Learned from The Right Time Show How to Advance Reproductive
Health During COVID-19 and Beyond.” Cambridge, MA: Mathematica, 2022.
5 Zolna, M.R., and J.J. Frost. “Publicly Funded Family Planning Clinics in 2015: Patterns and Trends in Service Delivery Practices and Protocols.” New York: Guttmacher Institute, 2016. Available at https://www.guttmacher.org/report/publicly-funded-family-planning-clinic-survey-2015.
6 Chew, L.D., K.A. Bradley, and E.J. Boyko. “Brief Questions to Identify Patients with Inadequate Health Literacy.” Family Medicine, vol. 36, no. 8, 2004, pp. 588–94.
7 National Academies of Sciences, Engineering, and Medicine. 2022. Measuring Sex, Gender Identity, and Sexual Orientation. Washington, DC: The National Academies Press. https://doi.org/10.17226/26424.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Mathematica Report Template |
| Author | Sharon Clark |
| File Modified | 0000-00-00 |
| File Created | 2022-08-11 |