passbacks

1. Please explain why the outcomes evaluation and the process evaluation are not being conducted at the same time.

Process evaluations are used to document how well a program has been implemented; they are conducted periodically throughout the duration of a program. This type of evaluation is used to examine the operations of a program, including which activities are taking place, who is conducting the activities, and who is reached through the activities. Process evaluations assess whether inputs or resources have been allocated or mobilized and whether activities are being implemented as planned

Outcome evaluations are used to assess the impact of a program on the stated short-term, intermediate, and long-term objectives. This type of evaluation assesses what has occurred because of the program and whether the program has achieved its outcome objectives. Outcome evaluations should be conducted only when the program is mature enough to produce the intended outcome.

The current evaluation will assess the initial process of conceiving, designing, and initially implementing the NNTCQ. Thus, this initial process evaluation is limited in time, to the period from conception through a time point when it is presumed to be established and functioning.

We have chosen one year from date when the 800-QUIT-NOW number went live (November, 2004) as a reasonable period for the Initiative to have become established. By November, 2005, enough of the implementation will have occurred to conduct an initial process evaluation. Because the Initiative is a work in progress, NCI expects to conduct further process evaluation as the Initiative evolves. For example, for a variety of reasons, one of the major elements of the Initiative, a national promotions campaign, will not commence until late 2006. The next round of evaluation will involve assessing both the process and impact of this campaign.

The consensus of the evaluation team and the expert panel is that it will take awhile for the effects of a broad-based public health initiative such as the NNTCQ to occur in the population at a level that can be detected through appropriate data collection tools. Specifically, it is critical to wait until the initial promotion campaign has occurred before attempting the initial outcome evaluation. The current plan is for this initial outcome evaluation to occur in the summer of 2007. Further outcome evaluation are also planned to assess the intermediate and long-term effects.

In this sense, NCI will be fulfilling the implication of OMB’s question, since the further process evaluations will be occurring in parallel with outcome evaluations. Process evaluation findings will be disseminated to grantees to make mid-course changes and programmatic improvements.

2. Confidentiality: It seems unlikely that the Privacy Act applies to this collection since individuals are reporting about their organizations not providing personal information about themselves. Does NIH have any other statutory authority to protect the confidentiality of this information? If not, then the pledge to respondents that you will keep the information confidential needs to be changed.

The applicable statutory authority comes from the authorities regarding the establishment of the National Institutes of Health, its general authority to conduct and fund research and to provide training assistance, and its general authority to maintain records in connection with these and its other functions (42 U.S.C. 203, 241, 289l-1 and 44 U.S.C. 3101), and Section 301 and 493 of the Public Health Service Act.

Under this authority, NIH maintains the umbrella system of records called Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD.

This system of records is an umbrella system comprising separate sets of records located either in the organizations responsible for conducting evaluations or at the sites of programs or activities under evaluation. Locations include Public Health Service (PHS) facilities, or facilities of contractors of the PHS.

Individuals covered by this system are those who provide information or opinions that are useful in evaluating programs or activities of the PHS other persons who have participated in or benefited from PHS programs or activities; or other persons included in evaluation studies for purposes of comparison. Such individuals may include (1) participants in research studies; (2) applicants for and recipients of grants, fellowships, traineeships or other awards; (3) employees, experts and consultants; (4) members of advisory committees; (5) other researchers, health care professionals, or individuals who have or are at risk of developing diseases or conditions studied by PHS; (6) persons who provide feedback about the value or usefulness of information they receive about PHS programs, activities or research results; (7) persons who have received Doctorate level degrees from U.S. institutions; (8) persons who have worked or studied at U.S. institutions that receive(d) institutional support from PHS.

This umbrella system of records covers a varying number of separate sets of records used in different evaluation studies. The categories of records in each set depend on the type of program being evaluated and the specific purpose of the evaluation.

The routine uses of records maintained in the system are:

1. Disclosure may be made to HHS contractors and collaborating researchers, organizations, and State and local officials for the purpose of conducting evaluation studies or collecting, aggregating, processing or analyzing records used in evaluation studies. The recipients are required to protect the confidentiality of such records.

2. Disclosure may be made to organizations deemed qualified by the Secretary to carry out quality assessments, medical audits or utilization review.

3. Disclosure may be made to a congressional office from the record of an individual in response to an inquiry from the congressional office made at the request of that individual.

4. The Department may disclose information from this system of records to the Department of Justice, to court or other tribunal, or to another party before such tribunal, when a) HHS, or any component thereof; or b) any HHS employee in his or her official capacity; or c) any HHS employee in his or her individual capacity where the Department of Justice (or HHS, where it is authorized to do so) has agreed to represent the employee; or d) the United States or any agency thereof where HHS or any of its components, is a party to litigation or has an interest in such litigation, and HHS determines that the use of such records by the Department of Justice, the tribunal, or the other party is relevant and necessary to the litigation and would help in the effective representation of the governmental party, provided, however, that in each case, HHS determines that such disclosure is compatible with the purpose for which the records were collected.

3. Burden hours: the 50% skip rate seems a bit high to me. In the survey for attachment 2a, for example, I counted only about 5 questions that would be skipped depending on whether the state had a quitline in 2004.

It is important to distinguish between question skips that will occur based on the state quitline’s date of establishment and all the other skips. The first type of skip is represented in the two panels of Table A-12.2 and A-12.3. For example, Table A-12.2 corresponds to Attachment 2a. It shows there are 10 question items that will be skipped based on the 2004 cutoff (bearing in mind that each subitems in a stem-and-leaf question is really a separate question): 7 questions in main question #5, plus the 3 questions #18, 19, and 23. Likewise, Table A-12.3 shows that 119 questions (out of 301) will be skipped for Attachment 2b, based on the 2004 cutoff.

Most important, though, is the fact that the approximate 50% skip rate applies to the questions after this allowance is made for the major skip. As described in Section A.12, “The other skips in these two instruments, and all skips in the other three instruments, are based on individual question responses; since there can be no a priori determination of such skips, the estimates use a 50% skip rate for the number of such questions. If anything, more than 50% will be skipped, since the preponderance of skips occur as follow-on questions in table-type questions listing a large superset of activities, situations, etc., that may apply to quitlines in the aggregate, but only a few of which will apply to any given quitline, thereby skipping the follow-on questions for most items in the list.”

Questions 9 and 10 of Attachment 2b offer a simple illustration of this approach. These two numbered questions constitute 9 question items, as counted in Table A.12-3. If a respondent answered No to each of the 4 questions in the first column of number 9, that person would skip 5 of the 9 questions. If he or she answered “Yes” to one of these questions and “No” to the other 2, he or she would skip 3 of the 9. Since these lists are supersets of the types of activities across all states, but a given state is not likely to respond “yes” to more than a small subset of these item. Other types of skips are the more familiar type, where there is a single question followed by a question asked only for a certain response to the first question. All things considered, we estimated that the average respondent would skip 50% of the questions.

In consideration of OMB’s expressed concern, we have reduced the estimated skip rate from 50% to 40%. This increased the average instrument timings between 1 and 3 minutes, except for the NAQC informant instrument, which was unaffected, since it contains no skips. The effect of this change on burden and annualized costs to respondents is detailed in the tables below, with an overall increase in burden of 6.90 hours and increase in costs of $289.80.

Original Table A.12-1. Estimate of respondent hour burden for the Process Evaluation of the NNTCQ Initiative @ 50% skip

Type of respondent	Estimated number of respondents	Frequency of Response	Average hours per response*	Annual hour burden
State Tobacco Control Manager	51	1	1.00	51.00
State Quitline Administrator	51	1	1.00	51.00
State Quitline Service Provider	19	1	.75	14.25
State Quitline Partner	102	1	.50	51.00
NAQC Representative	5	1	. 50	2.50
Total				169.75

Revised Table A.12-1. Estimate of respondent hour burden for the Process Evaluation of the NNTCQ Initiative @ 40% skip

Type of respondent	Estimated number of respondents	Frequency of Response	Average hours per response*	Annual hour burden
State Tobacco Control Manager	51	1	1.03	52.70
State Quitline Administrator	51	1	1.05	53.55
State Quitline Service Provider	19	1	.80	15.20
State Quitline Partner	102	1	.52	52.70
NAQC Representative	5	1	. 50	2.50
Total				176.65

Original Table A.12-7. Annualized costs to respondents @ 50% skip

Type of respondent	Estimated number of respondents	Frequency of Response	Average hours per response	Hourly Wage Rate	Respondent Cost
State Tobacco Control Manager	51	1	1.00	$42	$2142.00
State Quitline Administrator	51	1	1.00	$42	$2142.00
State Quitline Service Provider	19	1	.75	$42	$598.50
State Quitline Partner	102	1	.50	$42	$2142.00
NAQC Representative	5	1	.50	$42	$105.00
Total					$7,129.50

Revised Table A.12-7. Annualized costs to respondents @ 40% skip

Type of respondent	Estimated number of respondents	Frequency of Response	Average hours per response	Hourly Wage Rate	Respondent Cost
State Tobacco Control Manager	51	1	1.03	$42	$2213.4
State Quitline Administrator	51	1	1.05	$42	$2249.10
State Quitline Service Provider	19	1	.80	$42	$638.40
State Quitline Partner	102	1	.52	$42	$2213.4
NAQC Representative	5	1	.50	$42	$105.00
Total					$7,419.30

4. Qualitative data collection

1. Has the NIH considered the benefits of a staggered information collection approach, whereby subsequent interview questions can be added or modified depending on the information collected in prior interviews? This is typically one of the biggest advantages of using qualitative methods rather than quantitative: the process is iterative.

We appreciate your mentioning the possibility of this approach. Upon further consideration, there are two principal reasons why we would not choose to adopt it. The principal reason is that we engaged in considerable discussions among the key federal and NAQC players in designing the evaluation plan and analytical plan for the process evaluation. We also sought input from an expert panel, some of whom are the types of stakeholders from whom we will obtain the structured information from the planned data collection. As a result, the evaluation plan and the data collection methods utilize a semi-structured approach. While we will collect some qualitative data, an equal or greater amount will be categorical data. We are confident that we have now identified the issues and the specific information needed to inform the process evaluation analysis and report and can proceed to obtain the primary data components through the planned data collection instruments and respondents.

The second reason is one of timeliness. The planning process incorporated the kind of deliberate, iterative approach described in this question. Having devoted the time for this approach up front, we are confident that we can obtain the full range of information within the data collection structure and instruments that we have developed. The instruments are varied and robust. One reason why we have assumed a fairly high skip rate within the instruments (see ICR Section A.12 and our response to Question 3 above) is that the instruments are wide-ranging in their topics, to capture the diversity of circumstances and situations that exist in the NNTCQ and state quitline environment. While certain topics or questions may not reply to a given respondent, the collection of topics and questions anticipates the variety of situations and responses. Thus, at this juncture, we feel that we can obtain sufficient interview data with the existing instruments. The approach as originally proposed will allow us to proceed expeditiously though the data collection process at this time.

2. Does NIH have any reason to believe that lower level staff may have divergent views from key informants who are managers? If so, should they be incorporated into the study?

For this evaluation, there is a need for a diversity of inputs which is reflected in the variety of respondent types proposed and corresponding instruments. Among the quitline provider and partner respondents, there is no preconceived expectation for the level, role, or title of appropriate respondents. We will rely on the organizations to nominate the best respondent, fully expecting that they will represent a wide range of levels and roles in their organizations. Further, our data collection plan does not propose lower level staff as a discrete respondent set for several reasons. The main reason is that the focus of the evaluation is the NNTCQ, not individual state quitlines. One concept that we have taken pains to clarify amongst ourselves and the stakeholders is that this evaluation is not an evaluation of tobacco cessation or of individual state quitlines. The mangers of state tobacco control programs and the state tobacco quitlines are the individuals who are most aware of NNTCQ and in a position to asses its specific effect on the quitline environment. Thus, while lower level staff in the states might indirectly experience some effects of the NNTCQ, it would be difficult for them to attribute their experiences to the NNTCQ, if they are aware of it at all. NNTCQ is by its nature a high-level initiative, laid over or enabling state quitlines. The evaluation looks at this overlay, not the quitlines themselves. Indeed, in designing the state tobacco control manager and the state quitline administrator instruments, we have been challenged to formulate questions that will prompt even the high level managers to focus on and distinguish the NNTCQ from the many other phenomena occurring in tobacco control and state quitlines. The evaluation expert panel specifically advised that these managers are the types of staff who are in a position to furnish the data to support the evaluation plan. Having said all this, we do not plan a doctrinaire approach to respondent identification for these interviews; we will not be driven by titles, but will accept the recommendation of each state as to the best person to respond to each one.

3. While the initial request to record responses should not be done on tape, once respondents have agreed to have their interviews tape recorded, they should be asked a second time so that their consent is itself audio-recorded for the record.

This is an excellent idea. We will modify the contact procedure to incorporate it.

4. Is there some reason why respondents are not being asked about their experiences with this program during the 2006 year?

As noted in our response to Question 1, the current evaluation will assess the initial process of conceiving, designing, and initially implementing the NNTCQ. Thus, this initial process evaluation is limited in time, to the period from conception through a time point when it is presumed to be established and functioning. We have chosen one year from date when the 800-QUIT-NOW number went live (November, 2004) as a reasonable period for the Initiative to have become established. By November, 2005, enough of the implementation had occurred to inform a process evaluation.

A second reason is that the interview data will be used in parallel with secondary data the NAQC obtained in annual survey of the 51 state tobacco quitlines. Only the 2004 and 2005 NAQC data is available for the evaluation analysis and report.

5. Since you are asking respondents about their experiences in 2004 and 2005, what are the issues regarding respondent recall?

We agree that there may be a recall issues for some respondents or about certain items. We have adopted several approaches to minimize this potential impact a much as possible.

One is that we will seek as respondents the persons who are most knowledgeable and were most intimately involved with the activities and functions interacted with the NNTCQ. As recommended by the expert panel, we will specifically seek to speak with the person who functioned as state tobacco control managers or quitline administrator during the evaluation period, even if that person no longer has that responsibility. As noted above, we will encourage and rely on the states to identify and recommend the most knowledgeable persons for these interviews. The people who fulfill these roles are typically intimately involved with the policy, planning, funding, and operations. Our questions should remain highly salient to them, even after some time.

Further, we are not collecting explicit, detailed financial or operating statistics. We are asking for the respondents general sense of what things were like before and after the Initiative, and the extent to which, in their best judgment, the Initiative was a factor in any differences. This type of general sense or assessment of how things went is likely to persist. Were we seeking highly detailed information, recall would be more of a concern.

5. Qualitative data analysis

1. What kind of analytical techniques will NIH use to analyze the interview data?

Our description of the plans for tabulation in ICR Section A.16.1 includes a description of the analytical techniques. These will largely consist of summary descriptions of program attributes, plus descriptive statistics based on frequency counts and crosstabulations of means, medians, ranges, etc.

We did not previously mention that we will utilize comparisons of the descriptive statistics at two time points before (2004) and after (2005) the launch of the Initiative. While this is not a formal pre/post design, since it is being developed retrospectively, it will allow us to characterize what changed. The various questions concerning respondents’ view of the degree to which the Initiative influenced the observed changes will not allow us to attribute strict causality, but they will allow us to make statements of a general nature about whether the outputs of the implementation process began to produce changes, such as quitline capacity increases or increased promotional activity, that are posited to lead to the formal outcomes to be measured in the outcome evaluation.

For convenience, we are including here the table from ICR Section A.16.1.

Key Informant Group Survey	Research Questions	Measures and Analysis
NAQC	How effectively was the Initiative developed and launched? How effective were the collaborations that occurred?	Descriptive summaries of agencies involved, resources provided. Comparison of resources, collaborations provided by stakeholders. Mean effectiveness ratings of collaborations.
State Quitline Service Provider, State Quitline Administrator	How was the telecommunications infrastructure for 1-800-QUIT-NOW developed and implemented? Perceived benefits and barriers of telecommunications?	Descriptive summaries of steps involved in planning and implementing. Comparison of perceived benefits and barriers pre and post Initiative and across stakeholder groups.
State Tobacco Control Manager, State Quitline Administrator	How was the Request for Application (RFA) process developed and implemented? How did supplemental funding influence states with existing and non-existing quitlines?	Comparison of state tobacco funding environment (total tobacco $, quitline $, etc.) pre and post Initiative funding. Descriptive summary of RFA process from both federal and state perspectives.
State Tobacco Control Manager, State Quitline Administrator	What promotion efforts were planned and implemented for 1-800-QUIT-NOW? To what extent did 1-800-QUIT-NOW appear in the media? What promotion efforts were planned and implemented for state quitline numbers?	Descriptive summaries of promotion activities planned and implemented. Percent of Initiative funding used for promotions compared across states, for states with and without existing quitlines. .
State Tobacco Control Manager	What types of technical assistance, training and communications were provided to states relating to Initiative? How effective?	Response distributions of the number and type of technical assistance (TA), training, & communications provided by federal government. Comparison of mean utility ratings for TA, training & communications provided to states. Descriptions of TA, training and communications.
State Tobacco Control Manager, State Quitline Administrator	To what extent was state capacity to deliver quitline services enhanced or maintained by the Initiative?	Comparison of quitline services, programs and operations across three categories of states, 1) states with quitlines < $200,000; 2) states with quitlines >= $200, 000; and 3) states with no quitlines prior to the Initiative.
State Tobacco Control Manager, State Quitline Partner, State Quitline Administrator	How did the Initiative influence regional and state partnerships?	Means, response distributions of the number and types of partners in existence as a result of the Initiative. Descriptions of the partnerships, including functions, financial relationships, and activities.

2. What provisions are in place to maximize inter-interviewer and inter-coder reliability?

The interviewers will be mid- to senior-level professional research staff who have been intimately involved in the design and development of the evaluation. As such, they will be intimately familiar with the intent, issues, and constructs covered by the interviews and by the various types of activities and situations the respondents are involved in. The number of interviewers will be very small, probably two or three, which further enhances the consistency of the interviewing process and recording of responses. Immediately prior to the start of interviewing, the interviewers and analysts will participate in a meeting to review each question of each instrument to ensure a common understanding, identify potential problems or divergences, and formulate common solutions. For each set of interviews, the evaluation contractor’s senior analyst will listen to the recording of the first two interviews completed by each interviewer, while reading the completed questionnaire form. She will note any deviation from the standard or differences between interviewers, and correct or reconcile them. This will apply to both the oral conduct of the interview and the recording of responses on the instrument.

The only coding that may occur is post-coding of open-ended items. This coding will be done by the two data analysts who will analyze the survey data. Since they formulated the analysis plan that underlies the questionnaire content, they have a direct understanding of the intent of the coding. They will work collaboratively on any coding.

The small number of observations for each instrument combines with the small number of staff who will collect and code the data to strictly limit the latitude for inter-interviewer and inter-code error.

6. Sensitivity of questions: though the questions being asked are not of a personally sensitive nature, aren’t respondents likely to feel that their ability to implement their own quitline programs is being evaluated at some level? Aren’t the responses therefore likely to be skewed towards the positive? If so, what are the limitations of this study and how will these be addressed?

As noted above, we are taking pains to make clear that the evaluation is not an evaluation of quitlines in general, let alone a specific state’s quitline. It is an evaluation of a federal initiative in which states, state quitlines, and other stakeholders are partners. The Initiative is designed to enable the implementation and enhancement of the state quitlines. The Initiative did not constrain how states designed, configured, or operate their quitlines. The CDC Initiative funding laid out only very broad suggestions for appropriate use of funds (e.g. increased promotions, enhancing types of services provided).

It is true that some of the questions for state-level managers address what they did or what happened in regard to their own quitline, but the repeated focus is on how such experience was influenced by or related to the Initiative, not on their performance.

Further, unrelated to this evaluation, the states are already accountable on a semi-annual basis to the CDC for how they actually used funds provided through the Initiative; the tenor of this evaluation is much more neutral. In that regard, most of the questions have no obvious “good” or “right” answer. Of course, since “more” equals “better” in most people’s minds, a respondent might presume that, for any question about whether something increased, the “good” answer is “yes.” We also make clear that the evaluation is being conducted by an independent evaluation organization, and they will be speaking to a researcher, not a representative of the federal government. As such, individual responses that could potential identify specific respondents will not be provided to government personnel.

NCI, CDC, and the evaluation contractor have had a variety of opportunities to present and discuss the NNTCQ and the evaluation to large groups of potential respondents. We have explained the purpose of the evaluation to them. Perhaps the greatest comfort derives from the candor of the state representatives, provider and partner representatives, and NAQC representatives in these group meetings. In a public forum where there is the greatest temptation to show oneself in the best light, these stakeholders were forthright about successes and failures, problems and barriers, solutions and needs. We fully expect this candor in the one-on-one interviews.

If we do find or sense limitations resulting from “satisficing” answers, we can use the independent, highly quantitative data from the 2004 and 2005 NAQC surveys as a gauge and corrective.

7. It sounds like NIH is planning to revise the surveys even after OMB has reviewed them and issued approval (see B4). Does NIH intend to request an 83C for non-substantive changes and ICR-Revisions for substantive changes?

We do not plan to revise the questionnaire. Expert reviewers’ comments have been received and most were general recommendations only. Thus, please consider the questionnaire final.

8. Please provide more information on the qualifications and training of the “research analysts” who will conduct the interviews for this study (B.2). It was also noted that you would be using “executive” telephone interviewing techniques (B.4); please provide more information about this.

The research analysts are staff members of the evaluation contractor. They will be drawn from the current staff assigned to the evaluation. They will be master’s level professionals, with academic credentials and qualitative/quantitative research experience in various disciplines, including public health program evaluation and tobacco research. They will also have personal experience interviewing administrators, program managers, public agency staff, health care providers, and so forth.

“Executive interviewing” is a term that originally applied to private sector research conducted for marketing, strategic planning, etc. When such research needs information from high-level managers in companies or the public agencies, it is not feasible to approach such respondents with the tightly-scripted interviewing methods used for surveys of the general public, using interviewers of widely varying skills, knowledge, and ability. Rather, the interviewers are polished professionals who can hold their own in conversation with high-level managers and are already knowledgeable of or will receive in-depth training on the substantive aspects of the survey topic and the general aspects of the industry, roles, and responsibilities of the respondents. Such interviewers use skill in arranging interviews with managers who are hard to reach and who have busy schedules. They are also adept at the diplomatic aspects of conducting the interview. Most importantly, because of their personal skills and greater knowledge of the topic area, the interviewing process not only permits but actually depends on their entering into a dialogue with each respondent, in order to fully elicit the range and depth of information needed for such research.

9. Has the North American Quitline Consortium formally endorsed this study? Will this be incorporated into scripts or advance letters? Also, it didn’t appear that the interviewing scripts were referring to the advance letters that were sent.

Yes, the NAQC has collaborated with NCI in developing the study and supports it. We are not aware that they have formally endorsed it, in a manner that would allow us to make public statements to that effect. However, NACQ does plan to issue their own communications about the study through their weekly emails, newsletters, and periodic teleconferences, offering both information about the study and encouragement to participate. NAQC managers can exercise their discretion in this regard, while a formal endorsement would likely require official approval of their board of directors.

In various forms of meetings and communications over the past year, NCI, CDC, and the evaluation contractor have also informed the respondent pool of state tobacco control managers, state quitline administrators, quitline providers, and NACQ staff of the data collection. We have proposed sending advance letters to the partner respondents, since these are very diverse and are not very likely to have been previously reached through the channels described above. If OMB prefers, we will reference this advance letter in the introductory script.

10. Although you didn’t plan to pretest the data collection procedures, have you pretested or do you plan to pretest the survey instruments with potential respondents?

We consider the expert review previously referenced to be the most effective method for pretesting these instruments. The evaluation Expert Panel includes a state tobacco control manager and a senior manager of a major quitline service provider. In addition, the Executive Director of NAQC reviewed the instrument. This approach combines the benefits of expert review with the perspective of the proposed respondent groups. In addition, the members of the evaluation team from NCI, CDC, and the evaluation contractor have been working closely with the universe of respondents in the regular conduct of the NNTCQ Initiative and the development of the evaluation. We are aware of the near-unique circumstances of each of the states, their various managers, and their partners and providers.