SSB_TTC Survey_2-8-2011_JH

Supporting Statement B For:

Generic Submission for Technology Transfer Center (TTC)

External Customer Satisfaction Surveys (NCI)

February, 2011

John D. Hewes, Ph.D.

Technology Transfer Specialist

Technology Transfer Center

National Cancer Institute

6120 Executive Blvd., MSC 7181

Suite 450

Rockville, MD 20852

Phone: (301) 435-3121

Fax: (301) 402-2117

E-mail: hewesj@mail.nih.gov

Table of Contents

B. Statistical Methods 1

B.1 Respondent Universe and Sampling Methods 1

B.2 Procedures for the Collection of Information 6

B.3 Methods to Maximize Response Rates and Deal with Nonresponse 8

B.4 Test of Procedures or Methods to be Undertaken 11

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 12

List of Attachments for Supporting Statements A and B

Attachment 1: GEN IC - TechnOlogy Transfer center (TTC) External Customer CONSENT FORM AND Satisfaction Survey (NCI)

Attachment 2: ADVANCE NOTICE postal and e-mail Letter

Attachment 3: Followup E-mail

attachment 4: Reminder to participate e-mail

attachment 5: Additional Reminder to participate e-mail

attachment 6: final Reminder to participate e-mail

attachment 7: List of advisory Committee members

attachment 8: nih privacy officer’s letter

attachment 9: Office of Human Subjects Research (OHSR) exemption

attachment 10: Pre-Test FOLLOW-UP TELEPHONE INTERVIEW

AttACHMENT 11: Privacy Impact ASSessment (PIA)

B. Statistical Methods

B.1 Respondent Universe and Sampling Methods

The universe of potential survey participants consists of companies that conduct research and development in biomedical applications: pharmaceutical and biotechnology therapeutics, diagnostics, prognostics, medical devices, and software and are listed in the Technology Transfer Center Database. The survey universe (target population) includes company personnel who have utilized the services of the National Cancer Institute (NCI) Technology Transfer Center (TTC), “users” and “non-users” of TTC services, and those who have inquired about using TTC services, “inquirers.” The sex and race/ethnic composition of the respondent universe is not germane to the survey.

In proposing a sampling plan for the surveys, three sampling approaches were considered: a simple random sample, a stratified random sample, and a full population sample. These approaches are discussed below.

Simple Random Sample

There are 2,052 foundations, non-profit organizations, and for-profit companies that comprise the population of interest annually. A simple random sample of these companies is likely to retain the companies’ characteristics in the same proportions as the population as a whole within normal sampling variability. This method assumes that response rates would be equal across company characteristics, e.g., representatives of large companies are just as likely to respond as representatives of small companies, and there is no known influence of other company characteristics that would unduly influence the response rates.

In order to achieve item responses with a 90% level of confidence within a ± 3% margin of error, a sample size of 551 respondents would be required. Given that the response rate achieved in the pre-test was 80%, we assume a similar response rate in the population survey. Therefore, the first wave of sampling would “invite” approximately 690 company representatives in order to achieve the target sample size of n=551. If an 80% response rate is not achieved from the first sampling wave, a second sampling wave would be initiated. The number of new invitees would be calculated based on the number of respondents from the first wave with the same target sample of n=551, but now assuming a response rate based on the response rate achieved in the first sampling wave. For example if 500 company representatives responded during the first sampling wave (response rate=72%), an additional 51 respondents would be required to achieve the target of 551 company respondents. Therefore, a new random sample of 71 companies would be selected and invited to participate in the surveys, now assuming a 72% response rate, in order to achieve the desired target sample size.

TTC is interested in receiving input from three groups—users of TTC services, non-users of TTC services, and inquirers about TTC services. As shown in Table 1 (below), it is estimated that the size of the inquirer group is will be very small. In the past, there have been twice as many users as non-users. A serious disadvantage of drawing a simple random sample is that non-users will be underrepresented and inquirers may be seriously underrepresented. Since the size of the inquirer group is already very small, a further reduction in the size of this group would seriously limit the analyses that can be performed with that group.

Stratified Random Sample

In order to ensure samples that exactly mirror the population on some characteristic, a stratified random sampling procedure with the strata reflecting the population characteristic of interest may be utilized. As an illustration of this procedure, we have chosen the variable Category of User as the characteristic we are interested in mirroring the population of companies. Table 1 shows the strata and corresponding sample sizes for the population.

Table 1. Universe Covered by the Technology Transfer Center External Customer Satisfaction Surveys
Category of User of TTC Services	Number of Entities in the Universe	Target Sample Size	Number of Invitees in First Wave
Users of TTC Services	1,394	489	612
Non-Users of TTC Services	627	343	429
Inquirers about TTC Services	31	30	31
Total	2,052	862	1,072

The desired response estimation precision of 90% c.i. with ± 3% margin of error will be retained to illustrate differences in required sample sizes between this approach and the simple random sampling approach presented above. The desired precision is applied to each of the strata in order to obtain the target sample sizes within each strata (shown in the third column of Table 1). Similarly, assuming an 80% response rate as before, the number of required company representatives within each of the strata who would be invited to participate in the surveys is shown in the last column of Table 1 above.

As illustrated in Table 1, the target sample size for the stratified sampling approach (n=862) is nearly 60% larger than the target sample size required in the simple random sampling approach (n=551) with corresponding increases in the number of company representatives who would need to be invited to achieve these targets. The trade-off in insurance that the sample will maintain the proportion of companies within each of the strata is the increased target sample size required. This procedure could be applied to any of the variables of interest (e.g., public versus privately held companies) and target samples would be calculated as above. Note that the total number of Invitees is more than half of the population. In addition, utilization of a stratified random sample could increase the need for some nonresponse weights adjustment or some adjustments due to having differential response rates in the subgroups¹.

Population Survey

In situations where the population of interest is relatively small, one may choose to invite everyone included in the population to participate in the surveys. This is common in satisfaction surveys², particularly if the universe is relatively small as is the case with the Technology Transfer Center External Customer Satisfaction Surveys. These factors (relatively small population and stated purpose of survey), in conjunction with the data acquisition methodology i.e. web-based, recommends that a representative from each of the companies in the known population be invited to participate in the surveys. The on-line data capture methodology reduces the burden to individual responders and facilitates the data acquisition and data analysis process.

This methodology does not alter the assumption of an overall 80% response rate, providing a target sample of 1,642 company respondents for analysis. Confidence intervals would not be applicable to item responses as the universe of respondents is captured. The focus in this approach would be on the invitation process – providing multiple invitations to potential responders in order to maximize the response rate. This differs somewhat from the first two approaches where subsequent sampling waves are introduced in order to achieve the 80% response rate along with possible multiple invitations. The population survey methodology is less resource intensive in this case as the population is relatively small and choosing second or third wave random samples would not be required.

Analytically, the population survey method provides a larger sample thus increasing statistical power of the proposed analyses. In some instances the increase in power may be minimal (e.g., two-way classification tables) but for possible multivariate procedures the increased sample size could significantly increase statistical power and reduce the number of cells with small (or zero) sample sizes. Another significant advantage to the population survey approach lies in the value of open-ended (including “Other”) responses in customer satisfaction surveys. The larger sample size increases the probability of obtaining more of the positive and negative feedback responses that could inform change in, for example, how the TTC promotes its capabilities and how it promotes collaborations³ with outside companies that might not be captured in the smaller, random samples of company respondents. In addition due to the nature of the population, in analyses usually no weighting procedure is needed⁴. It is possible; however, that non-response bias could occur if non-responders to the survey share a particular characteristic or characteristics different from the group of survey responders. Non-response bias potentially increases as the overall response rate decreases; however, non-response bias can occur even if the targeted response rate of 80% (or higher) is achieved.  Therefore, characteristics of companies that fail to respond to the surveys will be compared to the characteristics of companies that do respond regardless of the achieved response rate. Key company characteristics include: company size, private vs. public ownership, location of parent company. In addition, a respondent characteristic - position in the company - will be compared between responders and non-responders. If responders and non-responders are nearly equally distributed on these characteristics than there is no evidence of bias and no statistical adjustments would be required. If, however, there are significant differences on any of the company or respondent characteristics believed to potentially influence responses to items on the survey, weighting techniques will be employed to adjust for the bias in survey responses.

In summary, given the three methods discussed above, the population survey method will be used for the TTC surveys.

B.2 Procedures for the Collection of Information

a. Survey Procedures

These are web-based surveys utilizing the Technology Transfer Center (TTC) External Customer Consent Form and Satisfaction Surveys (Attachment 1) that will be administered several times over a period of three years. Typically, each participant in the study will receive a postal advance notice and invitation followed by an e-mail advance notice and invitation to participate in the surveys (Attachment 2) from the NCI Project Officer that will explain the purpose of the survey, provide information about the confidentiality of their responses, and ask them to participate in the survey. One week later each participant will receive an e-mail invitation to participate sent by the Contractor (Attachment 3). This e-mail will contain a secure URL and password to access the web-based Technology Transfer Center External Customer Satisfaction Surveys. Two weeks after the e-mail invitation to participate has been sent, an e-mail “reminder to participate” (Attachment 4) will be sent by the Contractor to all survey non-respondents. An additional reminder to participate e-mail (Attachment 5) will be sent to non-responders one week prior to the closing date of the survey. A final reminder will be sent one day prior to the close of the survey (Attachment 6). Details of the announcement, invitations, and reminders (if used) will be included in the justification memos for subsequent generic IC sub-studies under this clearance.

b. Rationale for Sample Size

The population universe of 2,052 companies will be used for the TTC surveys for the following reasons:

1. With customer satisfaction surveys, it is customary to sample the population if possible;

2. Utilizing the entire population increases the sample size, reducing the likelihood of small/zero cells and increasing statistical power for subsequent analyses;

3. This method is less resource intensive during the surveying process;

4. Surveying the population increases the likelihood of obtaining open-ended comments that are often the key to customer satisfaction surveys; and

5. The data capture method i.e. on-line, facilitates data storage for purposes of analysis is not affected by sample size.

3. Quality Control.

The Contractor chosen for this study will establish and maintain quality control procedures to ensure standardization and a high level of quality of data collection and processing. The Contractor will maintain a written log of all decisions that affect study design, conduct, or analysis. Survey responses will be recorded electronically by the survey website. The Contractor will monitor performance of the data monitoring activities, especially with regard to response rates and completeness of acquired data. The Contractor will monitor response rates on a daily basis and download response data weekly while the survey is active. Each week’s data will be compared with the previous week’s data to identify any discrepancies. Once the survey has closed, manual spot-checks will be performed to ensure that results have been compiled and downloaded accurately.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

Several methods will be utilized to maximize response rates. The initial postal and e-mail invitations will come from the National Cancer Institute (NCI)—a source known and respected by the intended audience. Potential respondents’ participation will be requested in advance. Clear instructions will be provided on how to complete and submit the survey. Based on the results of the pre-test described below, the survey is designed for the intended audience and is easy to read and follow. Finally, the Contractor will send the follow-up invitation and reminder e-mails via the NIH e-mail accounts to increase credibility and decrease the likelihood of the e-mails ending up in the recipients’ junk mail folders.

The data collection method will be a web-based survey. The use of web-based technology for this survey was selected to reduce respondent burden, reduce costs, and improve data quality. Answering a set of questions via a web-based survey requires half as much time as answering the same set of questions via a telephone survey⁵. Data collection using web-based technology can improve data quality since validation checks can be incorporated into the instrument. In addition, because data are entered electronically, errors in data entry and coding are avoided. Many consumers now prefer the convenience of web-based surveys because they can be completed at a comfortable pace and at a time convenient for the respondent.

Two disadvantages sometimes ascribed to web-based surveys are lower response rates and issues related to reliability and validity of data. Response rates for paper based and CATI surveys have been rapidly falling over the last two decades⁶ . Conversely, response rates for web-based surveys have been growing with increasing internet usage among the general public⁷. Research comparing response rates among web-based, mail, and telephone surveys has traditionally found that response rates are generally lower for web-based surveys than for mail or telephone surveys; however, a meta-analysis of more recent studies has found that on average, web-based surveys yield only an 11% lower response rate compared to other modes (95% confidence interval 6%-15%) and that response rates are greatly increased when using targeted e-mail invitations⁸. In addition, response rates have been found to be greater among highly educated individuals such as our survey population⁹. With regard to the reliability and validity of data, web-based surveys are superior to computer assisted telephone interviews (CATI) in terms of their ability to moderate the effect of the interviewer on influencing social desirable responses¹⁰. Studies comparing methodologies have found that web-based surveys had the lowest rate of missing data items and CATI the highest¹¹. Web-based surveys are noted by many studies for delivering better verbatim responses than CATI surveys. Several studies argue that the quality of data collected through web-based surveys is generally higher than through CATI due to the respondent’s ability to re-read questions, elimination of time constraints or pressure, and completion at convenient times¹². Studies using external data verification have found that web-based surveys have the highest levels of reporting accuracy and CATI the worst¹³. Based on the literature cited, we are confident that the response rate will be adequate and the data collected will be reliable and valid.

The types of respondents will be managers and executives from companies that conduct biomedical research and development in therapeutics, diagnostics, prognostics, medical devices, and software. Procedures for follow-up of initial non-responders are described in Section B2a. Since this data collection effort is a customer satisfaction survey it is anticipated that the accuracy and reliability of the information will be more than adequate for the intended uses of the data. The face validity of the initial survey instrument was confirmed by the pre-test follow-up telephone interviews.

The initial generic survey contains 38 questions. Depending upon the respondent’s answers and the skip patterns built into the survey, a respondent may be asked to answer between 10 and 34 questions. Based on the pre-test, it is expected that the survey will take from 15 to 20 minutes to complete depending upon the number of questions the respondent is asked to answer. The survey will collect the following types of information:

• Respondent/company information: respondent’s current position in the company; company type; location of parent company; number of employees.

• Strategic direction of the company: whether they develop strategic technology collaborations; types of strategic technology collaborations; research collaborations formed within the last two years; research collaborations anticipated in the next two years; factors important to selecting a research collaborator; methods for finding new research collaborators; types of research collaborations developed within the past two years; state of research and development of research collaborations formed in the past two years; collaborations formed with off-shore companies, government laboratories, institutes, and/or universities; reasons for seeking off-shore collaborations.

• Experience with the NCI Technology Transfer Center (TTC) services: level of familiarity with NCI technology transfer services; how they learned about the NCI TTC; factors leading to not partnering with NIH researchers; factors leading to partnering with NIH; types of agreement completed; level of satisfaction with the length of time required to negotiate the agreement; feedback process; level of satisfaction with various aspects of the NCI technology transfer process; how they locate NIH research partners for potential collaborations; types of information the respondent would like to receive from the NCI TTC; preferred method of receiving NCI TTC information; additional TTC services that would be useful to meet the company’s technology transfer needs.

The basic formula (number of completed surveys/number of people contacted) will be used to calculate response rate. Based on the 80% response rate achieved for the pre-test and the fact that potential survey respondents are external customers who are managers and executives from companies that have a vested interest in biomedical and clinical research and a capability to pursue collaborations with NIH researchers, we project that 80% of the eligible 2,052 organizations will participate in this survey.

B.4 Test of Procedures or Methods to be Undertaken

A pre-test of eight respondents was conducted to ensure that the web-based survey was clear, well-designed and easy to use. The potential pre-test respondents received an invitation letter followed by an invitation e-mail from the NCI Project Officer inviting them to participate in a 2-stage process—first to complete and submit the web-based survey and then to participate in a brief telephone interview to discuss their experience with the web-based survey. Interview questions addressed accessibility and navigation; ease of comprehension and relevance; usability; and acceptability. The Pre-test Follow-up Telephone Interview protocol is included as Attachment 10. The initial survey was revised based on information learned from the pre-test and subsequent surveys may be altered slightly based on information from the first survey.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Individuals consulted on statistical design:

• Benmei Liu, PhD, Mathematical Statistician, Statistical Methodology and Applications Branch, Surveillance Research Program, Division of Cancer Control and Population Sciences, NCI. Phone: 301-594-9114

• Frederick Snyder, ABD, Statistician, NOVA Research Company; Phone: 301-986-1891

Contractor staff members of NOVA Research Company and The Madrillon Group Inc. will collect and analyze the data for NCI. Individuals who will analyze the data include:

Mary. C. Dufour, MD, MPH, Senior Vice President, The Madrillon Group Inc.

Jack E. Scott, ScD, Vice President for Evaluation, The Madrillon Group Inc.

Marietta Damon, PhD, Evaluation Specialist, The Madrillon Group Inc.

Frederick Snyder, ABD, Statistician, NOVA Research Company

National Cancer Institute and Contractor investigators and statisticians and Advisory Committee members (Attachment 7) have reviewed the data collection plan. The data collected will be available for use in analyses that are proposed and approved in the future. No additional consultation is planned.

1 Wholey, JS, Hatry, HP, Newcomer, KB. (2004) Handbook of Practical Program Evaluation. (2^nd Ed) Jossey-Bass, A Wiley Imprint, San Francisco, CA. 2004.

2 Fowler, FJ. (2009) Survey Research Methods 4^th edition. Sage Publications, Thousand Oaks CA.

3For the purpose of this survey, collaborations are defined as strategic alliances for co-development of technologies but not licensing or contractual relationships.

4 Wholey et al., 2004

5 Van Gelder M.M.H.J., Bretveld RW, and Roeleveld N. (2010). Web-based Questionnaires: The Future in Epidemiology? American Journal of Epidemiology Advance Access published September 29, 2010, doi: 10.1093/aje/kwq291

6 Curtin R, Presser S, Singer E. (2005). Changes in Telephone Survey Nonresponse over the Past Quarter Century. Public Opinion Quarterly 69(1):87-98.

7 Kaplowitz M D*, Hadlock T D, Levine R. (2004). A Comparison of Web and Mail Survey Response Rates. Public Opinion Quarterly 68(1):94–101

8 Manfreda, K L, Bosnjak, M, Berzelak, J, Haas, I, Vehovar, V. (2008). Web Surveys Versus Other Survey Modes: A Meta-analysis Comparing Response Rrates. International Journal of Market Research; 50(1):79-104.

9 Braithwaite D, Emery J, De Lusignan S, et al. (2003). Using the Internet to conduct surveys of health professionals: a valid alternative? Fam Pract. 20(5):545–551.

10 Kreuter F, Presser S, Tourangeau R. (2008). Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity. Public Opinion Quarterly 72(5):847-865; doi:10.1093/poq/nfn063

11 ibid

12 Kwak N, & Radler B. (2002). A Comparison Between Mail and Web Surveys: Response Pattern, Respondent Profile, and Data Quality. Journal of Official Statistics 18: 257-273.

13 Tourangeau R, Yan T. (2007). Sensitive Questions in Surveys. Psychological Bulletin 133:859–83.

3