Attachment B10 - Pilot Test Results

Attachment B10: Pilot Test Results

Note: The instruments included in this OMB package have been revised and incorporate the pilot test results. The question numbers correspond with the instruments pilot tested.

GUIDE SURVEY
Average Time to Complete = 14 minutes
Feedback	Changes Made to Instrument
General Comment: Consider whether geography will matter – Do you want to ask location of practice (urban, suburban, rural, Northeast, Midwest, etc)	Added two questions: (1) State where work and (2) work setting.
Question 5. Does not allow for dual degree. Include MPH in Masters category.	Added “check all that apply” and included MPH as an example.
Question 6. Include option to “check all that apply”.	Added “check all that apply”.
Question 7. Include option to “check all that apply”; two government organization options	Added “check all that apply” and removed one government organization.
Question 9. Trainings an odd word choice. Do you mean educational seminars or presentations (like Grand Rounds) or CME events?	Modified response options (e.g., removed Trainings, added educational seminar & presentations and CME events).
Question 10. Add option “Read most of the publication”	Added option “Read most of the publication”.
Question 12. Spell out what ePSS acronym stands for, since this is first time it’s used	Spelled out ePSS.
Question 13: Not sure how PS website option applies here. Do you mean to cite the web version of the pocket guide? This would make more sense.	Added explanation to indicate which version being asked about.
Question 16. This question would benefit from some categories to check (e.g., academic research, prof education, pt care, patient education, institutional/practice guideline devt) OR the categories you have in #27	Added response options from #27.
Question 18. Questions seem redundant. (i.e., up-to-date & contains new info, useful & provides info can use in my work)	No change made. Purpose requires level of detail.
Question 19. “Satisfaction” seems odd ..or is key ? How useful or valuable is the content to your work?	No change made. Need global assessment of respondents’ satisfaction with product content.
Question 20. Possibly consider a checklist with categories based on sections in the Guide (Summary, Rating, Clin Consids, Recs for Children, Immun Scheds, etc)	Added response options.
Question 22. Questions seem redundant	No change made. Purpose requires level of detail.
Question 26. Add “once or twice a year” as an option	Added option.
Question 27. Suggest condensing #27 and #16 or eliminating one altogether.	Modified question 16 and removed question 27.
Question 28. Is there a way to combine this with #26 in a mini-table?	Combined questions 26 & 28 into a table.
Question 29. Spell out what you mean by “apply the info” a little bit more (e.g., prev’n counseling, immune schedules for indiv pts)	Provided clarification.
If answered “no” to 32, 33-35 could be skipped	Added skip pattern.
Questions 33 & 34. Should have a NA choice for non-clinicians	Added N/A response option.
Questions 43 & 45. Can this be combined with #43 into one question	Underlined word to highlight differences between the questions. Kept both questions.
Also received feedback about some of the skip patterns and modified the instrument accordingly.
Incorporated relevant feedback from other instruments.

ePSS SURVEY
Average Time to Complete: 11 minutes
Feedback	Changes Made to Instrument
General Comment: It might be easier if you grouped questions by product vs. having them interspersed (back and forth)	Revised order of questions - grouped questions by product.
General Comment: Just say “ePSS” instead of “ePSS for PDA and/or ePSS for Web”	Changed text to read “ePSS”.
Question 5. Add “check all that apply	Added “check all that apply”.
Question 6. Might you use Physician MD and Osteopathic Physician (Physician DO?) since both are physicians	Added MD and DO to response option.
Question 7. Write out “other, please specify”	Wrote out “please”.
Question 19. It’s difficult to keep switching back & forth between products; maybe ask what % of time do you access ePSS using each method. If use PDA, answer xx –xx, if use web answer yy -yy	Revised order of questions - grouped questions by product.
Question 24. Add “If you use the ePSS for PDA or web” before “how often do you use it at the point of clin decisionmkg”	Skip pattern that was added eliminates the need for language change.
Question 25. Highlight “apply” Question 26. Highlight “refer to”	Underline added.
	Underline added.
Incorporated relevant feedback from other instruments.

WEBSITE SURVEY
Average Time to Complete: 12 minutes
Feedback		Changes Made to Instrument
General Comment: Avoid repetition of questions	Removed selected questions to minimize repetition.
Question 5. Are you asking the highest education achieved or all degrees received?	Added “check all that apply”.
Question 15. Choices seem repetitive (ie, useful & contains valuable info; rel to profl needs & provides info I can use in work)	Reduced questions.
Question 19. Limit # of options; last 5 options have same intent	Reduced questions.
Question 24. Ask this earlier in survey; change order of options 1,5,6,7,8,2,3,4	Re-formatted question and revised response options.
Question 26. Sentence does not flow	Revised question.
Question 28. Remove “not sure”…either aware or unaware. How can one be unsure if they are unaware?	Kept to give respondents the option.
Question 40. Change “practices” to “recommendations/ guidelines”	Revised question.
Question 43. “easy or difficult” 2 questions in one	Revised question. Response options provide a range of options from “very easy” to “very difficult”
Incorporated relevant feedback from other instruments.

INTERVIEW GUIDE
Feedback	Changes to Instrument
General Comment: There was some concern about the length of the instrument (i.e., too many questions)	No changes were made as these are semi-structured interviews and the interviewer can reduce the number of items if the respondent has included the answers in other questions or wishes to end the interview.
Question 5. Don’t like phrase “to attend to”	Revised question.
Question 10. Probe: “How can AHRQ interest others in using the Guide?” instead of current wording	Revised question to include recommended probe.
Question 16. “If so, how did you learn that updates were available?”	Revised question to include recommended language.