0990-CER_Portfolio Part B 6May2011

Supporting Statement for OMB Clearance of Assessment of American Recovery and Reinvestment Act (ARRA) Comparative Effectiveness Research (ACERE)

Section B: Collection of Information Employing Statistical Methods

Report

May 6, 2011

Project Officer:

Kate Goodrich, M.D., MHS

Medical Officer

Office of the Assistant Secretary for Planning and Evaluation

U.S. Department of Health and Human Services

200 Independence Avenue, SW

Washington, DC 20201

CONTENTS

Respondent Universe and Sampling Methods 2

Procedures for Collecting the Information 6

Methods to Maximize Response Rates and Deal with Nonresponse 10

Tests of Procedures or Methods to be Undertaken 11

Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data 11

TABLES

B.1. Proposed Population and Sample Counts for SSLA Survey of Key Stakeholders 3

B.2. Fielding Schedule for the PI/PD Survey 7

B.3. Fielding Period Schedule for the Survey of Stakeholders 8

B.4. Individuals Responsible for Statistical Aspects and Data Collection and Analysis 12

LIST OF ATTACHMENTS

Attachment A: Piloting Summary

Attachment B: PSLA—Web-based survey of PIs and PDs.

Attachment C: PSLA—In-depth telephone interviews with PIs and PDs

Attachment D: SSLA—Web-based survey of three key stakeholder groups
in two rounds

Attachment E: SSLA—Focus groups with members of the general public
in two rounds

Attachment F: SSLA— In-depth telephone interviews with providers

Attachment G: SSLA— In-depth telephone interviews with health care organizations

Attachment H: SSLA— In-depth telephone interviews with patients/consumers

Attachment I: SSLA— In-depth telephone interviews with employers and payers

Attachment J: SSLA— In-depth telephone interviews with researchers

Attachment K: SSLA— In-depth telephone interviews with innovators

Attachment L: Invitation to participate and reminder notices for PIs and PDs

Attachment M: Invitation to participate and reminder notices for stakeholders

B. Collection of Information Employing Statistical Methods

The Assistant Secretary for Planning and Evaluation (ASPE) is requesting approval from the Office of Management and Budget (OMB) for the following data collection activities to support an evaluation and impact assessment of the American Recovery and Reinvestment Act of 2009 (ARRA) comparative effectiveness research (CER) portfolio:

• A web-based survey of principal investigators (PIs) and project directors (PDs) who have received ARRA funds to conduct CER;

• In-depth telephone interviews with PIs and PDs;

• A web-based survey conducted at two points in time (round 1 in Fall 2011 and round 2 in Fall 2012) with three groups of key stakeholders in CER: health care providers, health care administrators, and patients/consumers;

• Focus groups with members of the general public at two points in time (round 1 in Fall 2011 and round 2 in Fall 2012);

• In-depth telephone interviews with stakeholders of CER, including health care providers, health care administrators, patients/consumers, members of the general public, employers and payers, researchers, and developers of health innovations.

Overview of the Study Design. This project seeks to evaluate and assess the products and outcomes of ARRA-funded CER investments and the impacts of those investments on the priority topics recommended by the Institute of Medicine and on the categories and themes of the Federal Coordinating Council on Comparative Effectiveness Research (FCCCER) and Department of Health and Human Services (HHS) frameworks. The evaluation will also gauge the evolution in CER-related knowledge and skills, opinions and attitudes, and behaviors and experiences held by stakeholders and society in general and will ultimately draw lessons for future CER funding. The evaluation will incorporate data from new and existing sources and use multiple data collection methods over a two-year period to assess the broad array of CER-relevant federal programs and stakeholder and community perspectives.

The evaluation design consists of a mixed-methods approach for addressing the effectiveness of the ARRA CER portfolio in meeting its programmatic goals. The primary goals of this evaluation are to:

1. Conduct an initial assessment of the ARRA CER portfolio, cataloguing how CER funding was invested to achieve the vision of the FCCCER, and assessing initial impacts from the perspective of various stakeholders;

2. Lay the groundwork for future CER investments by identifying investment opportunities, evidence gaps and lessons learned, providing the tools for ongoing assessment, and providing recommendations for future investments.

The evaluation consists of two components (each with unique objectives) with data collection activities that involve public burden and for which clearance is requested: (1) a Project-Specific Level of Analysis (PSLA) and (2) a Societal/Stakeholder Level of Analysis (SSLA). The PSLA will identify the achievements and lessons learned from specific ARRA-funded CER projects, whereas the SSLA will undertake a broad study of knowledge and attitudes held by stakeholders and members of society in general. Each component is discussed in turn below.

Overview of the PSLA. The PSLA will provide information on the products, outputs, and outcomes of ARRA-funded CER projects, identify the factors that facilitate or limit project success, and determine whether there are systematic gaps in projects’ design, conduct, or dissemination that limit their value to decision makers. This level of analysis includes collecting data from the public via two methods: (1) a web-based survey of PIs and PDs, and (2) in-depth telephone interviews with PIs and PDs.

Overview of the SSLA. The SSLA will provide information on CER-relevant knowledge and skills; attitudes and opinions; and behaviors and experiences among key stakeholders and members of the general public. This component of the evaluation will obtain information from several groups, including members of the general public who have no direct involvement in CER, persons or groups who have a vested interest in clinical decisions and the evidence that supports those decisions, and individuals or groups directly involved in CER projects. This component will use three public data collection activities to collect information on knowledge and skills, attitudes and beliefs, and behaviors and experiences to understand attitudes toward CER and the processes that stakeholders use to engage in it. The SSLA activities for collecting data from the public include: (1) two rounds of a web-based survey of three key stakeholder groups: health care providers, health care administrators, and patients/consumers; (2) two rounds of focus groups with members of the general public; and (3) in-depth telephone interviews with stakeholders.

The SSLA survey and focus groups will each be conducted at two points in time in order to evaluate whether there is any change in the knowledge, attitudes, and behaviors of the sampling frame populations over time. The first round will begin around November 2011 and the second round will occur approximately one year later with different participants, using the same instruments as at round 1.

1. Respondent Universe and Sampling Methods

The primary goal of the sampling design is to support the collection of data to answer evaluation questions about the following two factors:

• the products, outputs, and outcomes of ARRA-funded CER projects

• the CER-relevant knowledge and skills, attitudes and opinions, and behaviors and experiences of key groups among those who have a vested interest in clinical decisions and the evidence that supports those decisions.

The sampling designs for the public data collection activities for the evaluation are described below.

PSLA – Web-based survey of principal investigators and project directors (PIs and PDs). The universe of all relevant PIs and PDs who are the grantees is not sufficiently large to allow for sampling. Thus, the entire universe of ARRA-funded CER grantees will be covered by the data collection. ASPE has contact information for each grantee. In addition to the approximately 430 ARRA-funded grantees, the sampling frame will consist of one additional researcher from each ARRA-funded CER project, as identified by the named PI or PD. The data collected from the survey will provide important details about projects that cannot be obtained through existing HHS documents or from other sources and has not been conducted previously. The estimated response rate for the grantees is 70 percent, and the estimated response rate for the approximately 300 additional researchers identified is 40 percent. The subject matter and purpose of the survey are salient to the universe of grantees. For these reasons, a high response rate to this survey is expected among this group. Since the additional researchers might not be as strongly connected to the funding agency, a lower response rate among this group is expected.

PSLA—In-depth telephone interviews with PIs and PDs. The evaluation will identify and conduct telephone interviews with up to 50 ARRA-funded PIs and PDs. The purpose of this data collection activity is to explore in greater depth the challenges, barriers, and limitations, as well as the successes and promise, of the ARRA CER portfolio from the perspective of PIs and PDs. The sampling frame consists of the entire universe of grantees and is the same as the sampling frame used for the web-based survey of PIs and PDs described above. The sample for these in-depth interviews will be selected purposively; interviewees will be chosen to reflect the diversity of the ARRA CER portfolio. To identify the projects to be investigated via these interviews, the chosen projects will represent a cross-section of the core categories and priority themes of the FCCCER strategic framework. To do this, the contractor will use a checklist of criteria to determine whether to consider a project as a candidate for closer examination. The criteria are both general, applying to all projects, and specific to projects in each core category. For example, general criteria include such factors as whether the project involved components of more than one core category or priority theme or whether there was stakeholder or decision maker involvement in the project. More specific criteria relevant to each core category include whether a project had a research component attached to it (for non-research activities) or whether a project established a new training program (for human and scientific capital projects). Because PIs and PDs have a vested interest in the subject matter, we expect they will respond at high rates. Thus 62 individuals will be selected in order to complete 50 interviews (a response rate of 80 percent).

SSLA—Round 1 and round 2 web-based survey of three key stakeholder groups. The stakeholder survey will provide data to answer questions about the CER-relevant knowledge and skills; attitudes and opinions; behaviors and experiences of key groups among those who have a vested interest in clinical decisions and the evidence that supports those decisions. The two rounds of data collection will be conducted with a different sample, so that each round represents a cross-section. The three key stakeholder groups include: (1) health care providers, (2) health care organizations, and (3) consumers. Table B.1 shows the respondent universe and sample size.

Table B.1. Proposed Population and Sample Counts for SSLA Survey of Key Stakeholders

Population	Sampling frame source	Approximate sampling frame size	Expected sample size per round
Health care providers	MMS1	707,000	600
Health care organizations	JCAHCO2	19,000	600
Patients/Consumers	MSG3	2,000,000	600

¹ MMS is Medical Marketing Systems

² JCAHCO is the Joint Commission on Accreditation of Health Care Organizations

³ MSG is Marketing Systems Group

For the three key stakeholder groups, we will collect data on knowledge of, attitudes toward, and experiences with CER both for general CER topics and for specific CER topic areas. Because the three key stakeholder groups consist of different types of individuals, the sampling frame and sample for each population will be obtained from a different source. Because the contractor will not have access to the sampling frames directly, each source will select a stratified or simple random sample according to the contractor’s specifications. ASPE’s contractor will request a sample size somewhat larger than the number expected to be needed to obtain the targeted number of completes. Random replicates of the sample will be released, and sample, in waves, will be released as needed, after observing response patterns during data collection. The target sample size is 600 respondents per round from each of the three key stakeholder groups (a total of 1,800 per round). The sampling frame for each stakeholder group is described in more detail below.

• Health Care Providers. The sample will be obtained from Medical Marketing Systems, Inc. (MMS). MMS purchases and pools lists of health care provider organizations. The providers that make up the respondent universe include physicians, nurse practitioners, and physician assistants. MMS purchases lists of providers from large membership organizations, such as the American Medical Association, and from state registries of health professionals, and pools this information into comprehensive lists. MMS collects demographic information about their sample members in addition to variables related to primary and secondary specialty and type of practice.

Once MMS pools the sampling frame together for the included providers, MMS will then select a stratified random sample of providers, with explicit strata defined by provider type (physician, advanced practice nurse, and physician assistant). We will also work with MMS to determine if they can “implicitly stratify” or sort the sampling frame by other characteristics within stratum before sampling. Each sampled individual is chosen entirely by chance and each member of the population has an equal chance of being included in the sample within stratum. The main benefit of stratified random sampling is that we can help ensure that the sample chosen is reflective of the stakeholder population. There are limited data on the characteristics of the national populations of different provider types, but we will know provider type, and should know specialty for physicians and perhaps geographic location for all providers. Because this is an online survey, the sampling frame will be restricted to people for whom MMS has an email address on file. Contact information to conduct telephone reminder calls to non-respondents will be included in the sample file; however, to protect the email privacy of its members, MMS does not provide the email addresses of its sample members. When email invitations to the surveys are ready to be delivered, MMS will mail them directly according to a schedule determined by the contractor project team.

• Health Care Organizations. The sample of health care organizations will be obtained from the Joint Commission on Accreditation of Health Care Organizations (JCAHCO). JCAHCO owns a comprehensive list of accredited health organizations; using this list is the most efficient way to identify a representative sample of organizations. The contractor will request that JCAHCO select a sample implicitly stratified by organization type and geographic region. JCAHCO accredits the following types of organizations: ambulatory care, behavioral health care, critical access hospitals, home care organizations, hospitals, laboratories, and long term care facilities. Identifying additional organizations not accredited by JCAHCO or not accredited at all may be preferable for generalizing to the national population of health care organizations at large. However this would entail enormous costs to acquire the sample and to screen for eligibility for the survey.

• Patients/Consumers. The patient/consumer sample includes individuals with chronic conditions. To obtain a sampling frame of email addresses for consumers, the contractor will use an internet panel. To draw a sample that can be deemed as representative of the population of interest, the panel must be large and the sample supplier must collect extensive demographic information about its sample members. The internet panel maintained by Marketing Systems Group (MSG) contains more than two million people and is extensively profiled for demographic information. MSG will provide a sample of people who are adults with chronic diseases. The sample from the online panel provided by MSG will also contain information on household composition, the presence of children in the household, and age of sample members. The contractor will request a probability sample of these patients, implicitly stratifying by one or more characteristics such as gender, age group, disease type, and geographic location. Unlike the provider and organization components of the SSLA, the contractor will contact the consumer sample members by email directly.

SSLA—Round 1 and round 2 focus groups with members of the general public. The evaluation will include six focus groups in each of two rounds in three large metropolitan areas to ensure geographic and demographic diversity: Cambridge/Boston, Massachusetts; Oakland/San Francisco, California; and Chicago, Illinois. The focus groups will examine the knowledge and skills, attitudes and opinions, and behaviors and experiences of the general public in CER. These three sites were selected for the demographic diversity they offer in order to ensure that a diverse group of individuals is represented at the focus groups. In addition, recruiting in large metropolitan areas is more likely to yield diversity with respect to experiences with the health care system.

Although results from focus groups cannot be used to generalize to a larger population, focus group participants should include people with a broad range of demographic and other characteristics that may be related to whether they understand, use, or are interested in CER. Thus recruitment for the focus groups will adequately represent people from different backgrounds in terms of race/ethnicity, gender, income level, education, self-assessed health status, and experiences with the health care system. In this case, representativeness does not mean that the makeup of the groups reflects the makeup of the population at large, given that the sample sizes are small and the samples are not selected using probability methods. Rather, it means that at least some of the group participants will fall into these key categories, to give a voice to people in that group. This can be controlled for by determining, before recruitment, what the makeup of the groups should be so that recruitment is targeted appropriately.

The focus groups will be conducted at professional focus group facilities. The sample for the focus groups will come from lists compiled by the focus group facility in each city where the focus groups will be held. When recruiting for the groups, the facility will use a script developed by the contractor that will convey to potential participants the purpose of the research, the legitimacy of the focus groups, and the $50 incentive for participation, and will screen for characteristics of interest. Each focus group will be conducted with 10 participants, for a total of 60 participants in each round, or 120 for both rounds. In order to recruit 60 participants, we expect to contact approximately 75 individuals, for a response rate of 80 percent.

SSLA—In-depth telephone interviews with stakeholders. The purpose of the in-depth stakeholder telephone interviews is to collect information on the three primary domains of interest--knowledge and skills, attitudes and beliefs, and behaviors and experiences—in an effort to understand attitudes toward CER and the processes stakeholders use to engage in it. The interviews will also follow up on issues raised in earlier data collection activities such as the first round of stakeholder surveys and focus groups with the general public. To examine differences in the three domains held by various stakeholder groups, this information will be collected from six stakeholder groups, namely: (1) health care providers, (2) health care organization administrators, (3) patients/consumers, (4) employers and payers, (5) researchers, and (6) developers of health innovations.

Up to 60 qualitative telephone interviews will be conducted, with approximately 10 stakeholders from each of the six stakeholder groups. For the three key stakeholder groups—providers, health care organizations, and consumers/patients—the survey sample pool will provide the source from which to contact a purposive sample of additional providers, patients/consumers, and health care organizations to participate in the telephone interviews.

For each of the additional three stakeholder groups—researchers, employers/payers, and innovators—the contractor will ask the relevant professional groups for the names of about 13 people (for researchers and innovators) to 20 people (for employers) to interview, along with contact information and job/position information. We expect researchers and innovators to have a vested interest in the topic of CER and to respond at higher rates than employers. For researchers, interview subjects will be obtained by contacting three organizations for recommendations: (1) the American Association of Medical Colleges, AAMC, a nonprofit that represents all 133 accredited U.S. medical schools, approximately 400 major teaching hospitals and health systems, and nearly 90 academic and scientific societies; (2) the Association of Academic Health Centers, AAHC, a nonprofit that supports academic health center leaders; and (3) AcademyHealth, a professional society that represents a broad community of health services researchers and health policy analysts. For employers and payers, the sample will first be split between the two groups (five interviews with employers and five with payers). For employers, interview participants will be obtained using the Dunn and Bradstreet database, which represents both large and small employers. For payers, interview subjects will include both state government employee health insurance managers and private health insurance companies, including capitated group practices such as Kaiser or Group Health. For state government employee health insurance managers, sample will be obtained by contacting the National Academy for State Health Policy. For innovators, interview subjects will be obtained by contacting the three largest representatives of health care intervention innovators: (1) Pharmaceutical Research and Manufacturers of America (PhRMA; drug and biologics manufacturers), and (2) Advanced Medical Technology Association (AdvaMed; device manufacturers).

2. Procedures for Collecting the Information

Each data collection activity in the evaluation will follow separate data collection procedures, described below.

Survey Data Collection. For both the PSLA survey of PIs and PDs and the SSLA survey of key stakeholders, the primary mode of data collection is through a web-based data collection instrument. The contractor will program, test, and implement the web-based survey instruments. The web instrument will offer the easiest means of providing data as it will be programmed to automatically skip questions that are not relevant to the respondent. The instrument will also allow respondents to complete the survey at a time that is convenient to them without the risk of losing a paper survey questionnaire. Since the instruments will automatically skip to the next appropriate question based on a respondent’s answers, it will also provide high quality data. To lessen respondent burden, in addition to the web instrument, participants may request a hard copy questionnaire or receive telephone assistance in completing the survey with the contractor’s facility liaison.

The data collection procedures for each of the two surveys being conducted as part of this evaluation are described below.

• PSLA web-based survey of PIs and PDs. The fielding period for this survey will be eight weeks, which should be sufficient to achieve a 70 percent response rate among named PIs (approximately 300 completed questionnaires) and a 40 percent rate among additional PIs (approximately 120 completed questionnaires). At the start of the field period, the contractor will send an email invitation to each PI or PD that will include a unique URL, username and password to access the web-based questionnaire. The unique username will allow tracking of who responded and who requires additional follow-up. The contractor will send email reminders (up to four) to nonrespondents beginning in the second week of the field period. Although all investigators will have email and internet access, some may prefer to complete the questionnaire by telephone. During reminder calls (up to two) in weeks 4 and 6, trained interviewers from the contractor’s facility will call investigators to remind them of the survey and offer to complete the survey by telephone. If needed, one reminder will be sent by mail in the fifth week of the fielding period. Table B.2 describes the fielding schedule for this survey.

Table B.2. Fielding Schedule for the PI/PD Survey

Week	Activity for Web-based Survey of PIs and PDs and their Nominees
	Named PI/PD	Nominees, Group 1	Nominees, Group 2	Nominees, Group 3	Nominees, Group 4
0	Advance letter from ASPE
1	Email invitation
2	Email reminder
3	Email reminder	Email invitation
4	Telephone reminder	Email reminder	Email invitation
5	Mail and email reminders	Telephone reminder	Email reminder	Email invitation
6	Telephone reminder	Email reminder	Telephone reminder	Email reminder
7	Email reminder from Project Officer	Email reminder from Project Officer	Email reminder from Project Officer	Email reminder from Project Officer	Email invitation
8	Final email reminder	Final email reminder	Final email reminder	Final email reminder	Email reminder

• SSLA web-based survey of key stakeholders. As with the PSLA survey, the key stakeholder survey will be administered online to maximize convenience and minimize burden. Since providers and administrators in health care organizations are very busy, the online survey will allow them to complete the survey in multiple sittings if they so choose. Some survey items will be common to all groups and some will be tailored to the relevant stakeholder perspective. For example, demographic items will be common to all sample members. Because patients, providers, and administrators in health care organizations have different roles in searching for CER-related information and use CER differently in decision making, questions about information seeking and decision making will vary by group.

The data collection protocol involves follow-up with sample members in several ways (Table B.3). Communication with all sample members will begin with an advance letter sent by postal mail. The advance letter will alert sample members to the survey and describe its purpose and the importance of their participation. It will also convey the legitimacy of the survey request. Within approximately one week, all sample members will then receive an email invitation to the survey. The mail will contain a web address to the online questionnaire and a unique username and password for sample members to log into the survey. The unique username will enable the contractor to track who responded and who requires additional follow-up. Non-respondents will receive up to two reminders by mail and up to three reminders by email to complete their survey. In addition to reminder mailings and the incentive payment, the contractor will conduct up to four calls to potential respondents. Trained interviewers at the contractor’s facility will place phone calls to non-respondents to encourage them to complete the survey.

Table B.3. Fielding Period Schedule for the Survey of Stakeholders

Week	Activity for Web-based Survey of Key Stakeholders
0	Advance letter from ASPE
1	Email invitation
2	Email reminder
3	Email reminder
4	Telephone reminder
5	Mail reminder, email reminder
6	Telephone reminder
7	ASPE mail reminder
8	Final email reminder

Focus Group Data Collection. For the SSLA focus groups with the general public, the focus group mode is a group discussion to be conducted in-person with a moderator. Because we expect relatively little knowledge or understanding of CER among the general public currently, a focus group is the most appropriate tool for uncovering attitudes of the general public about CER. The focus group protocol was developed with the understanding that knowledge among the general public about CER is relatively low. Because it is difficult to gauge people’s understanding under these circumstances, participants will be presented with information and vignettes or hypothetical scenarios. These scenarios will present treatment decisions with evidence-based options that have different pros and cons, and participants will be asked to react to them. This approach will examine how much people understand, their affective reactions toward the material, and their ability to engage with similar material in their own lives when making real decisions about their own health. Participants can choose not to answer any question they do not want to answer.

Although the focus groups will be conducted at professional focus group facilities, contractor staff will moderate them. A professional focus group facility provides the staff to greet participants as they arrive at the location and explain the focus group procedures to them. The discussion itself will take place in a room with a one-way mirror, allowing one or two observers to watch the groups without the group seeing them. The focus group moderator will follow the focus group protocol but will also ask unscripted follow-up questions if important insights are revealed during the discussion. The moderator will also make sure that everyone in the group has a chance to speak and assure participants that there are not right or wrong answers as we would like to hear a variety of viewpoints.

In-Depth Interview Data Collection. The PSLA component of the evaluation includes 50 in-depth interviews with PIs and PDs. The SSLA component of the evaluation includes 60 in-depth interviews among the following groups of stakeholders: (1) health care providers, (2) health care organization administrators, (3) patients/consumers, (4) employers and payers, (5) researchers, and (6) developers of health innovations. The contractor will mail each selected interviewee an email invitation for the interview, and will follow up with a telephone call if necessary. For each interview, there will be an interviewer and a note taker on the telephone. After the interviews are complete, responses will be coded electronically using the Atlas.ti program, which will facilitate analysis of the data. To reduce burden on respondents, contractor staff will schedule interviews at times most convenient to the sample members.

Degree of accuracy needed for the purpose described in the justification. The expected sample sizes for the PSLA and SSLA web surveys will allow us to detect differences between subgroups of respondents as indicated in the table below. For both surveys, this table assumes 80 percent power to detect differences of these sizes at the 0.05 significance level, when the outcome is a proportion of approximately 0.5. The sample sizes shown for the provider stakeholder group are actually “effective sample sizes,” which adjust the actual sample sizes to account for the design effect (that is, the increase in the variance) due to having somewhat different selection probabilities across the three groups: providers, health care organizations, and patients/consumers. For purposes of our analysis, we have assumed six comparisons across these SSLA stakeholder groups. At the broadest level, we will compare providers to health care organizations, providers to patients/consumers, and health care organizations to patients/consumers. Within the provider stakeholder group, we will compare physicians to nurse practitioners and physician assistants. Within the patient/consumer group, we will compare patients with chronic conditions to caregivers and parents to other caregivers. For the PSLA survey, we have assumed two analytic comparisons. First, a comparison of outcomes of activities funded by grants to those funded by contracts. Second, a comparison of projects that were funded as a follow-on to existing research versus those that were new, original research, assuming that each group made up about half the sample.

Tabe B.4 Minimum Detectable Differences for SSLA and PSLA Web Surveys

Web Survey	Subgroup 1	Subgroup 2	Effective Sample Size Subgroup 1	Effective Sample Size Subgroup 2	MDD for Proportion of 0.5
SSLA	Provider	HC Organization	553	600	.083
	Provider	Patient	553	600	.083
	HC Organization	Patient	600	600	.081
	Physician	NP and PA	400	196	.122
	Chronic Pts1	Caregivers	200	200	.140
	Parents	Caregivers	200	200	.140
PSLA	Grants	Contracts	301	120	.152
	Follow-on Project	New Research	211	210	.137

Source: Mathematica Policy Research

Note: HC = Health Care. NP = Nurse Practitioner. PA = Physician Assistant. These calculations assume a small design effect within the provider sample due to the oversampling of the nurse practitioners and physician assistants. We assume a two-sided test of the differences in proportions that are near 0.50 with alpha equal to .05 and 80 percent power. For example, if the true underlying difference in a proportion between providers and organizations is .09, then we would detect such a difference as significant at least 80 percent of the time, given these sample sizes.

¹ Patients with chronic conditions who responded to the survey on their own.

Data Security. The contractor has a secure server for online data collection utilizing its existing and continuously tested web-survey infrastructure. This infrastructure features the use of HTTPS (secure socket, encrypted) data communication; authentication (login and password); firewalls; and multiple layers of servers, all implemented on a mixture of platforms and systems to minimize vulnerability to security breaches.

Hosting on an HTTPS site insures that data are transmitted using 128-bit encryption so that transmissions that are intercepted by unauthorized users cannot be read as plain text. This security measure is an addition to standard password authentication that precludes unauthorized users from accessing the web application.

The contractor has established data security plans for the handling of all data during all phases of survey execution and data processing for the surveys that it conducts. Its existing plans meet the requirements of U.S. federal government agencies and are continually reviewed in the light of new government requirements and survey needs. Such security is based on (1) exacting company policy promulgated by the highest corporate officers in consultation with systems staff and outside consultants, (2) a secure systems infrastructure that is continually monitored and evaluated with respect to security risks, and (3) secure work practices of an informed staff that take all necessary precautions when dealing with confidential data.

Confidential data are kept in study-specific folders to which only the minimum number of staff members are allowed access. All survey data as well as qualitative data that are electronically coded are backed up continuously and preserved on secure media.

3. Methods to Maximize Response Rates and Deal with Nonresponse

The data collection activities involving focus groups and in-depth interviews are not based on probability samples and are not meant to represent anyone other than the respondents, and therefore a response rate does not apply to these activities. For the two survey data collection activities that form part of this evaluation, the methods to maximize response rates are described below.

ASPE expects the web-based survey of PIs and PDs to achieve at least a 70 percent response rate among PI and PD grantees, and a 40 percent response rate among additional researchers named by the grantee. The subject matter and purpose of the survey are highly salient to the universe of grantees. For these reasons, a high response rate to this survey is expected among this group. Since the additional grantees are not as strongly connected to the funding agency, we expect a lower response rate among this group. These estimates are also based on the contractor’s experience conducting web-based surveys with PIs on other studies.

Because the expected response rate for the PSLA web survey is less than 80 percent, we will analyze nonresponse patterns using whatever relevant information is known about both respondents and nonrespondents. We will compare the characteristics of respondents, nonrespondents, and the total attempted sample, both unweighted and weighted, to evaluate the risk for nonresponse bias of estimates, which cannot be directly measured.

In addition, numerous methods will be used to encourage response. The initial message to PIs and PDs will be produced on ASPE letterhead to heighten the salience of the survey request and to activate the norm of reciprocity. The initial contact to key stakeholders will come from ASPE to highlight the importance of the survey through the survey sponsorship.

The contractor will send email reminders (up to four) to nonrespondents beginning in the second week of the fielding period. The contractor will also place reminder calls to nonrespondents. Although all investigators will have email and internet access, some may prefer to complete the questionnaire by telephone. During reminder calls (up to two) in weeks 4 and 6, trained interviewers from the contractor’s facility will call investigators to remind them of the survey and offer to complete it by telephone. While the primary purpose of the follow-up calls will be to encourage participation, the callers will also offer to assist sample members over the phone to complete the survey. If needed, a mailed reminder will be sent in the fifth week of the fielding period. Tables B.2 and B.3 show the fielding schedule for the two surveys and their respective nonresponse follow-up activities.

For the PSLA survey with PIs and PDs, no incentive will be offered for completing the survey. For the SSLA survey with key stakeholders, respondents will receive a $20 incentive for participating. Incentives have been found to have a statistically and substantively significant positive effect on response rates among medical providers (Kellerman and Herold, 2001; Thorpe et al 2009; Field et al 2002), organizational representatives (Simsek and Veiga, 2001), and the general public (Goritz 2006).

Because the expected response rate for the SSLA survey is about 50 percent, we will analyze nonresponse patterns using whatever relevant information is known about both respondents and nonrespondents. We will compare the characteristics of respondents, nonrespondents, and the total attempted sample, both unweighted and weighted, to evaluate the risk for nonresponse bias of estimates, which cannot be directly measured.

The survey team will review frequencies early and often to swiftly identify any nonresponse patterns. If the data are submitted by mail or fax, they will be keyed into the web instrument and the frequencies checked in the same way as web-collected data.

4. Tests of Procedures or Methods to be Undertaken

The following instruments were pretested in January 2011: the survey questionnaire for the PSLA web-based survey of PIs and PDs and the interview protocols for the SSLA in-depth interviews with stakeholders. The protocol for the PSLA in-depth interview with PIs and PDs was pretested in February 2011. The survey questionnaire for the SSLA web-based survey of stakeholders was pretested January-March 2011. The protocol for the SSLA focus groups with the general public was not pretested as it is not feasible to pretest a focus group protocol without conducting the focus group itself.

The pretest activities confirmed that the information being requested in the in-person interviews and web-based surveys for the current evaluation is reasonable, clearly stated in coherent unambiguous language, and collected in the least burdensome way possible. The survey questionnaires and in-depth interview protocols were each pretested with fewer than 10 respondents to learn about problems respondents might experience in providing the requested information and to make appropriate changes to the questionnaires and protocols. Pretest responses and comments to the survey questionnaires were collected by mail to emulate the self-administration that will be used for the survey. Contractor staff followed up with pretest respondents by telephone to learn their reactions and determine how to improve language. Pretest responses to the in-depth interviews were collected by telephone to emulate the in-depth interview procedures. The pretest also established the average interview length, currently estimated at 20 minutes per PI/PD survey and 15 minutes per stakeholder survey. The results of the pretest were used to revise the questionnaire.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

No individuals outside the evaluation project were consulted on statistical aspects of the design. ASPE has contracted with Mathematica Policy Research to conduct this evaluation. Table B.4 identifies the individuals at this organization who will be responsible for collecting and analyzing the data. The Project Officer for the contract providing funding for the evaluation, Kate Goodrich, will be responsible for receiving and approving all contract deliverables. Her contact information is also included in Table B.4.

Table B.4. Individuals Responsible for Statistical Aspects and Data Collection and Analysis

Name	Title (Project Role)	Organizational Affiliation and Address	Phone Number
Melanie Au	Researcher (Data collection design, analysis)	Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024	202-264-3459
Danna Basson	Survey Researcher (Data collection design, data collection management)	Mathematica Policy Research 505 14th Street Suite 800 Oakland, CA 94612	510-830-3713
Mark Brinkley	Senior Systems Analyst (Data collection programming)	Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024	202-484-4243
Barbara Carlson	Senior Statistician and Associate Director of Statistical Services (Sampling Statistics)	955 Massachusetts Ave., Suite 801 Cambridge, MA 02139	617-674-8372
Arnold Chen	Senior Researcher (Analysis)	600 Alexander Park Princeton, NJ 08540	609-275-2336
Leah Einfeld	Program Associate (Project Management)	Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024	202-484-5265
Dominick Esposito	Senior Researcher (Data collection design, analysis)	600 Alexander Park Princeton, NJ 08540	609-275-2358
Christopher Fleming	Research Analyst (Data collection design, analysis)	Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024	202-554-7537
Sarah Forrestal	Survey Researcher (Data collection design, data collection management)	111 East Wacker Dr., Suite 920 Chicago, IL 60601	312-994-1017
Mindy Hu	Survey Specialist (Data collection design, data collection management)	Mathematica Policy Research 505 14th Street Suite 800 Oakland, CA 94612	510-830-3710
Brice Overcash	Research Analyst (Data collection design, analysis)	600 Alexander Park Princeton, NJ 08540	609-945-3366
Stephanie Peterson	Research Analyst (Data collection design, analysis)	Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024	202-484-4691
Eugene Rich	Senior Fellow (Data collection design, analysis)	Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024	202-250-3544
Sunyna Williams	Senior Researcher (Data collection design, analysis)	Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024	202-250-3514
Hong Zhang	Senior Programmer (Data collection programming)	600 Alexander Park Princeton, NJ 08540	609-716-4545
Kate Goodrich	Medical Officer in the Office of the Assistant Secretary for Planning and Evaluation(ASPE) (Project Officer)	200 Independence Ave. S.W. Washington DC 20201	202-690-7213