Form 4 Attachment G: CDS Connect – Leader

Attachment G

Form Approved
OMB No. 0935-XXXX
Exp. Date XX/XX/20XX

CDS Connect – Leader

This protocol will be used for AHRQ Project Officers and the MITRE Project Team.

Introduction/Consent

• Introduce members of group.

• Thank you very much for your time today.

• NORC at the University of Chicago is a not-for-profit research organization, and we are working with the Agency for Healthcare Research and Quality (AHRQ) as the independent evaluator of the Patient Centered Outcomes Research (PCOR) Clinical Decision Support (CDS) Initiative.

• I am going to be leading this interview, but others may chime in with follow-up questions.

• Just a few things before we get started.

  • Your participation is voluntary and you can conclude the discussion at any time. We are interested in your opinions – it is completely okay for you not to answer any questions that you do not want to. There are no wrong answers to the questions I will ask.

  • We will not attribute anything you have to say as coming from you personally. We will keep your name confidential in any summaries or reports we make available to AHRQ or the public.

    • You are also free to make comments “off record” in which case we will only consider them as background.

  • We have scheduled this meeting to last [x] minutes. If you need to stop for any reason, that is fine. We know you are busy and may schedule a follow up interview or e-mail you to address any unanswered questions. We appreciate your participation.

• We have a member of our team from NORC taking notes so we can write our reports, and we would like to make an audio recording to help make sure we capture everything correctly. The notes and recording will only be used by NORC to write our reports.

• Do you offer your consent to participate in the interview, and are you okay with us recording our conversation?

• Do you have any questions before we begin?

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average XX minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

Opening

• Could you please start by introducing yourself, and your organization?

• What is your personal or organizational experience with CDS development and/or implementation?

• What has been your role in CDS Connect?

  • What is your background in the field of CDS development?

• How did you hear about AHRQ’s PCOR CDS Initiative and the CDS Connect component of it?

• What is your understanding of PC CDS?

• To what extent do you think the PCOR CDS Initiative has made a contribution to the development of this type of CDS?

  • What about the CDS Connect project specifically?

Overview and Context

[For MITRE only]

• Please tell us your personal and MITRE's organizational experience with CDS development that led to your work on the CDS Connect initiative?

  • How does this work relate to any other work being undertaken by MITRE through the CMS Alliance to Modernize Healthcare, which technically operates CDS Connect?

• Please walk us through a high-level timeline of the development of CDS Connect Authoring Tool and repository?

• What was the process like between AHRQ and MITRE setting expectations for the design of CDS Connect Authoring Tool and the repository? And the timeline?

• Are you familiar with other CDS repositories? If so, which ones?

  • How are they similar/different to CDS Connect?

  • What, if any, is the relationship between CDS connect and other repositories/initiatives?

[For AHRQ leaders]

• Were you involved in the development of the RFP for CDS Connect? If yes, can you talk about why AHRQ felt it was important to fund the development of a repository like CDS Connect?

• Was there prior work that AHRQ was involved with/funded which influenced the concept of CDS Connect?

Workgroups

• Based on our understanding, CDS Connect held monthly workgroups including one related to cholesterol management and another related to technical requirements. Were there any additional workgroups that you considered convening? Why did these others not get off the ground?

• Would you please describe why you engaged these workgroup(s)?

  • What were your roles/objectives?

  • Can you describe why the two workgroups merged into one after the first year? How did this impact original objectives of the two workgroups?

• Please describe the stakeholders who were represented in these workgroup(s)? Were there any stakeholders not engaged who you felt should have been? Were there any stakeholders you did not initially expect or plan to be engaged who were?

• Did patients/advocates/caregivers play a role in these workgroup(s)? If yes, what role did they play?

• What were the barriers and facilitators to engaging stakeholder in and accomplishing the goals of the workgroup(s)?

Repository/Artifacts

• Please describe the development phases of the CDS Connect repository?

  • What enhancements were made over time? Why were these made?

  • Can you describe the main features of the CDS Connect repository?

  • How do these features promote the use of shareable, standards-based, CDS artifacts?

  • How did the Workgroups contribute to the design of the CDS Connect repository?

• Our understanding is that one of the primary goals of CDS Connect Repository was to build a trusted platform to disseminate evidence-based CDS. How did you define or think about "trust" in the context of the CDS Connect Repository?

  • Does this change when you think about the perspective of the Consumer vs Contributor?

• What features or characteristics of the Repository help engender trust of Contributors and Consumers?

• What issues or concerns have you faced with trust since the development of CDS Connect Repository?

• To what extent do you believe that at this time is the Repository and the artifacts within the Repository are trusted for the successful dissemination of CDS? Please describe why or why not.

• Did you provide any TA to people who contacted you through the website and/or participated in the workgroups? What kinds of questions did people have, and who or what kinds of people were asking these questions?

• How did you, as a manager of CDS Connect, review artifacts for inclusion in the CDS repository?

  • Is this the same for all submissions, or does it differ based on the source?

  • Are there common reasons why an artifact is or is not accepted?

• What impact does making shareable standards-based, CDS artifacts freely available have on a commercial CDS marketplace?

  • What impact do you hope it will have?

  • Conversely, how does the EHR vendor marketplace influence the use of shareable standards-based, CDS artifacts from CDS Connect?

• Do you track artifacts moving from Knowledge Level 1 to Level 4? How?

  • To what extent is it important to document the process of translating artifacts from L1 to L4?

  • Can/should this process be standardized?

• Were there any particular lessons learned or recommendations that you could draw from these experiences related to developing CDS or a repository of CDS?

Pilots

• We have read quite a bit about the three CDS pilots including each pilot’s final reports. In your words would you please describe the three CDS pilots that were developed under the CDS Connect umbrella?

  • How did you determine the focus of these three pilots?

• How did your experience with the pilots inform the development of the Repository and Authoring Tool?

Patient/Caregivers/Patient Advocates

• Do individuals who contribute artifacts indicate whether they have incorporated patient or caregiver preferences in the development of the artifact? If yes, how so?

• To your knowledge, have any of the artifacts incorporated patient or caregiver preferences?

  • How do the artifact(s) incorporate patient or caregiver preferences?

• From your perspective, what are the barriers and facilitators to engaging patients or caregivers in CDS?
  
  

Contributors

• What are the primary reasons for submitting artifacts to CDS Connect?

• Why would someone choose to contribute an artifact to CDS Connect versus another CDS repository?

• Are some types of stakeholders more or less likely to contribute to CDS Connect? Why?

• Would you please describe barriers related to IP in the sharing and publication of CDS?

• Are there any other reasons why someone might hesitate to contribute artifacts to CDS Connect?
  
  

Consumers

• What are the objectives of consumer who access resources from CDS Connect?

• What are the main reasons someone might choose to use an artifact from CDS Connect versus another CDS repository?

• Are there certain types of stakeholders who are more or less likely to use artifacts from CDS Connect? Why?

• To what extent have you tried to track, formally or informally, consumers of the CDS connect artifacts?

  • Would you be able to share any names of individuals or organizations that have used the authoring tool or any of the artifacts?

• What are the barriers and facilitators to consumers accessing the resources from CDS Connect?

• What are the barriers and facilitators to consumers implementing the CDS artifacts?

• Have you tried to enhance the use of the artifacts in any way? Please describe.

Authoring Tool

• Would you please describe how the concept for the authoring tool was created?

  • Did you engage with any external (to MITRE) stakeholders in the conceptualization process?

• Would you please describe the technical process for developing the authoring tool?

  • Did you engage with any external (to MITRE) stakeholders in the development process?

• What updates were made to the Authoring Tool since the first release?

  • Are there any other planned updates?

  • Who was involved in these decisions?

• We understand that the CDS Authoring tool supports the creation of CDS logic that are exported as HL7 CQL CDS artifacts and that use the HL7 FHIR data model. Are there other ways that the CDS authoring tool supports the creation of standards-based CDS?

GitHub Open Source Resources

• Can you describe the four resources that CDS Connect has made available on GitHub?

• Would you please describe how you came to the decision to make open source resources related to Authoring Tool and CDS Connect API available on GitHub?

  • What barriers did you encounter in sharing these resources on GitHub?

  • Who is most likely to be interested in using these resources? Why?

  • Have you ever been contacted by someone who used these resources? What did you learn from them about their use of the resources?

Impact

• From your perspective, have the resources developed through CDS Connect changed the CDS field? If yes, please describe the ways the field has changed.

• To date, what would you consider the primary accomplishments of CDS Connect?

• What have been the top challenges?

• Please share with us recommendations for standing up a CDS repository, or the ongoing management of the CDS Connect repository?

• To what extent do you feel that CDS Connect is disrupting the field of CDS development and encouraging more developers to make shareable, standards-based, interoperable CDS? Why?

• What are the major barriers and facilitators for the adoption of the CDS Connect tools on the field of PC CDS development?

• Were the activities and resources of CDS Connect informed by other non-AHRQ collaborative initiatives working to advance the field of evidence-based CDS development?

• To your knowledge did the activities and/or resources of CDS Connect feed into or inform other non-AHRQ collaborative initiatives working to advance the field of evidence-based CDS?

Sustainability

• Is there a plan to continue supporting CDS Connect, including the repository, Authoring Tool and GitHub resources, after the end of MITRE's contract?

  • Who will be responsible for reviewing and testing artifacts?

  • Who will be responsible for updating the website with new artifacts?

  • Who will be updating the Authoring Tool to align with new standards?