instructions for completing the Expert Panel Review Rating form

National Evaluation of the Comprehensive Technical Assistance Centers

Instructions for Completing the Expert Panel Review

Rating Form

Purpose

As part of the Comprehensive Center evaluation, expert review panelists will independently rate the technical quality of a sample of projects selected from each Center. Technical quality will be assessed along three dimensions (Exhibit 1):

• Demonstrated use of the appropriate documented knowledge base

• Fidelity of application of the knowledge base to the products and services provided

• Clear and effective delivery

For each project sampled, the Centers will assemble materials (e.g., documents produced, conference proceedings, policy briefs, etc) that represent the various products, services and activities associated with this effort. The Centers will also complete a cover sheet that provides relevant contextual information. Together, these materials will be considered as a single package (representing the project) and will be rated using a standard scoring rubric (Exhibit 2). The rubric provides uniform, objective criteria for rating technical quality and defines each scale point for each dimension. Reviewers will assign a score to each dimension, using a 5-point rating scale, according to the definitions and examples provided.

Each project will be rated by three reviewers. Reviewers will not be asked to assign an overall score to the project. Instead, the evaluation team will compute an aggregate project-level score for each reviewer by simply averaging their dimension scores. We will also compute an overall quality score for each project by averaging the three reviewers’ aggregate scores.

Reviewing Materials and Completing the Rating Form

Panelists should review all items associated with a sampled project when considering the various dimensions to be rated. Using the scoring rubric provided, the reviewers will assign a score to each dimension and document on the rating form (Exhibit 3) the basis for their ratings, including the specific artifacts on which their score is based.

To guide the review process, we provide the following instructions:

• Review the whole packet. The unit of analysis for this review is the project, not the individual products, services, or activities. As such, you should consider all materials in the packet (including the cover sheet) when assigning scores. Be aware that not all data sources will be relevant to each of the technical quality dimensions. Alternately, some data sources will provide information for assessing multiple quality dimensions.

• Note relevant information/evidence on the rating form as you go along. Each dimension score should be supported by documentation in the “Notes” section of the rating form. As you review the materials in the packet, please record specific examples (including the artifacts in which they were found) that demonstrate the presence (or absence) of that particular dimension of technical quality. For example, if a literature review is included in one of the documents, the document name should be listed in the “Notes” section under Dimension 1 on the rating form, along with your assessment of how accurate, thorough, and balanced the review is. Similarly, if there is no mention of the knowledge base in any of the materials, this should be noted.

• Assign scores based on the rubric provided. The rubric specifies the standards for each score point within each dimension. After reviewing all of the materials, you should assign a score of “1”, “2”, “3”, “4”, or “5” based on the definitions provided in the rubric. Remember that the basis for your ratings (including the specific artifacts) must be documented on the rating form. If, after reviewing all the data sources for a project, you still believe that there isn’t enough information on which to base a rating, assign a score of “0” to that dimension.

Exhibit 1. Dimensions of Technical Quality

• Demonstrated use of the appropriate documented knowledge base
  
  

• Do materials indicate that an accurate, comprehensive, and balanced review was conducted of the applicable knowledge base (empirical research, literature review, laws and regulations, professional wisdom) for the particular content area?

• Are citations provided and sources noted for the knowledge base? Is the research that forms the basis for the project’s products and services derived from the latest generation of findings, based on relevant interventions, and from studies with appropriate designs?

• Is the legal or regulatory background informing the project’s products and services accurately interpreted and applicable for the content area?

• If professional wisdom or other non-empirical evidence forms the basis for findings or recommendations in products and services, is that clearly noted in materials?

• Are limitations in the knowledge base for the content area adequately identified and acknowledged in the products and services provided and a range of viable alternatives presented for consideration by clients?

• Fidelity of application of the knowledge base to the products and services provided
  
  

• Are the products, assistance, and advice provided by the Center for this project consistent with accurate and appropriate application of the core findings from the available knowledge base?

• Is there clarity and consistency between the products developed and assistance and advice given and the level of rigor and certainty in the available knowledge base?

• Was greater weight given to findings derived from the most rigorous and consistent research?

• Are recommendations in products and services that are based on a weak research base, limited legal or regulatory guidance, or primarily non-empirical professional wisdom appropriately tempered?

• In the absence of a strong knowledge base in the content area, are a range of viable alternatives suggested for consideration by clients?

• Clear and effective delivery
  
  

• Are the findings, recommendations, or assistance provided by the Center in conjunction with this project well-written, well-organized, and relevant—clear, accurate, applicable and complete—for their intended audience so as to support the use and implementation of their content in priority areas?

Exhibit 2. Draft Scoring Rubrics: Technical Quality

Technical Quality Dimension 1 Demonstrated use of the appropriate documented knowledge base: Materials for this project indicate that an accurate, comprehensive, and balanced review was conducted of the applicable knowledge base (empirical research, literature review, laws and regulations, professional wisdom) for the particular content area. Citations are provided and sources noted for the knowledge base. Research is derived from the latest generation of findings, based on relevant interventions, and from studies with appropriate designs. Legal or regulatory background informing the project’s products and services is accurately interpreted and applicable for the content area. Materials clearly note where professional wisdom or other non-empirical evidence forms the basis for findings or recommendations. Limitations in the knowledge base for the content area are adequately identified and acknowledged in products and services provided.
5 “Very High Quality”	Project materials contain citations and reference all or almost all of the most important and relevant research studies, legal guidance, or other available information on this topic. Where available, prominence is always given to research that is scientifically based, reflects current professional thinking, and that used appropriate designs, methodologies, and measures. The knowledge base is accurately interpreted in all cases. Where the knowledge base is limited, appropriate limitations and caveats are always included in presentation of findings, recommendations, and other information.
4 “High Quality”	Project materials contain citations and reference many of the most important and relevant research studies, legal guidance, or other available information on this topic. Where available, prominence is usually given to research that is scientifically based, reflects current professional thinking, and that used appropriate designs, methodologies, and measures. The knowledge base is accurately interpreted in most cases. Where the knowledge base is limited, appropriate limitations and caveats are usually included in presentation of findings, recommendations, and other information.
3 “Moderate Quality”	Project materials contain some citations and reference some of the most important and relevant research studies, legal guidance, or other available information on this topic. Where available, prominence is sometimes given to research that is scientifically based, reflects current professional thinking, and that used appropriate designs, methodologies, and measures. The knowledge base is accurately interpreted in some cases. Where the knowledge base is limited, appropriate limitations and caveats are sometimes included in presentation of findings, recommendations, and other information.
2 “Low Quality”	Project materials contain limited citations and reference a limited set of the most important and relevant research studies, legal guidance, or other available information on this topic. Although available, little prominence is given to research that is scientifically based. Research that is cited does not typically reflect current professional thinking or does not use appropriate designs, methodologies, and measures. The knowledge base is not accurately interpreted in most cases. Where the knowledge base is limited, appropriate limitations and caveats are not typically included in presentation of findings, recommendations, and other information.
1 “Very Low Quality”	Project materials contain no or almost no citations and do not reference the most important and relevant research studies, legal guidance, or other available information on this topic. Although it is available, no prominence is given to research that is scientifically based. Research that is cited does not reflect current professional thinking or does not use appropriate designs, methodologies, and measures. The knowledge base is not accurately interpreted. Where the knowledge base is limited, appropriate limitations and caveats are not included in presentation of findings, recommendations, and other information.
0 “Not enough evidence to judge”	The project materials submitted for review do not contain enough information to determine a rating on knowledge of the relevant literature and research.

Technical Quality Dimension 2 Fidelity of Application of the Knowledge Base in Products and Assistance Provided: Materials for this project indicate that products, assistance, and advice provided to clients are consistent with the core findings from the available knowledge base. There is clear and consistent treatment in the products, advice and assistance provided in relation to the level of rigor and certainty in the available knowledge base. Preference is given to providing products, assistance, and advice based on scientifically-based research and clear legal and regulatory guidance. When only a weak research or knowledge base is available, products, assistance, and advice provided are appropriately tempered and professional wisdom is incorporated and identified in suggesting a range of viable alternatives for consideration by clients.
5 “Very High Quality”	Project materials indicate that products, assistance, and advice provided to clients is always based on findings from the available knowledge base. Preference is always given to scientifically-based research or clear legal and regulatory guidance to inform the products, assistance, and advice given. Where such scientifically-based research or clear legal and regulatory guidance is not available to inform assistance in the content area, appropriate limitations and caveats are always given when products, assistance and advice are provided and professional wisdom is always used and identified to suggest a range of viable alternatives for clients to consider.
4 “High Quality”	Project materials indicate that products, assistance, and advice provided to clients is usually based on findings from the available knowledge base. Preference is usually given to scientifically-based research or clear legal and regulatory guidance to inform the products, assistance, and advice given. Where such scientifically-based research or clear legal and regulatory guidance is not available to inform assistance in the content area, appropriate limitations and caveats are usually given when products, assistance and advice are provided and professional wisdom is usually used and identified to suggest a range of viable alternatives for clients to consider.
3 “Moderate Quality”	Project materials indicate that products, assistance, and advice provided to clients is sometimes based on findings from the available knowledge base. Preference is sometimes given to scientifically-based research or clear legal and regulatory guidance to inform the products, assistance, and advice given. Where such scientifically-based research or clear legal and regulatory guidance is not available to inform assistance in the content area, appropriate limitations and caveats is sometimes given when products, assistance and advice are provided and professional wisdom is sometimes used and identified to suggest a range of viable alternatives for clients to consider.
2 “Low Quality”	Project materials indicate that products, assistance, and advice provided to clients is infrequently based on findings from the available knowledge base. Preference is infrequently given to scientifically-based research or clear legal and regulatory guidance to inform the products, assistance, and advice given. Where such scientifically-based research or clear legal and regulatory guidance is not available to inform assistance in the content area, appropriate limitations and caveats is infrequently given when products, assistance and advice are provided. A range of possible viable alternatives is infrequently presented for clients to consider when scientifically-based research and clear regulatory and legal guidance is not available and professional wisdom is infrequently used.
1 “Very Low Quality”	Project materials indicate that products, assistance, and advice provided to clients is not based on findings from the available knowledge base. Preference is not given to scientifically-based research or clear legal and regulatory guidance to inform the products, assistance, and advice given. Where such scientifically-based research or clear legal and regulatory guidance is not available to inform assistance in the content area, appropriate limitations and caveats are not given when products, assistance and advice are provided. A range of possible viable alternatives is not presented for clients to consider when scientifically-based research is not available to inform assistance nor is professional wisdom used.
0 “Not enough evidence to judge”	The project materials submitted for review do not contain enough information to determine a rating on fidelity of application of the knowledge base in products and assistance provided.

Technical Quality Dimension 3 Clear and effective delivery of products and assistance: Materials for this project indicate that products, assistance, and advice provided to clients are well-written and well-organized—clear, accurate, applicable and complete—for the intended audience. The presentation of findings, information, recommendations or assistance as contained in materials is appropriately designed for the intended audience and supports the use and implementation of the content.
5 “Very High Quality”	Project materials indicate that the information conveyed in the products, assistance, and advice for this project is totally clear, accurate, and complete. Products and services for this project fully communicate information accurately. None of the most important findings or information from the knowledge base is omitted. Project products and services fully communicate relevant findings (or information from the knowledge base) effectively, in clear and accessible language. In the rater’s judgment, the information conveyed would be easily understood by the intended audience to support the use and implementation of the content in all cases.
4 “High Quality”	Project materials indicate that the information conveyed in the products, assistance, and advice for this project is mostly clear, accurate, and complete. Project products and services mostly communicate information accurately. Very little of the most important findings or information from the knowledge base is omitted. Most of the project’s products and services communicate relevant findings (or information from the knowledge base) effectively, in clear and accessible language. In the rater’s judgment, the information conveyed would be easily understood by the intended audience to support the use and implementation of the content in most cases.
3 “Moderate Quality”	Project materials indicate that the information conveyed in products, assistance, and advice for this project is sometimes clear, accurate, and complete. Project products and services sometimes communicate information accurately. Some of the most important findings or information from the knowledge base is omitted. Some of the project’s products and services communicate relevant findings (or information from the knowledge base) effectively, in clear and accessible language. In the rater’s judgment, the information conveyed would be easily understood by the intended audience to support the use and implementation of the content in only some cases.
2 “Low Quality”	Project materials indicate that the information conveyed in products, assistance, and advice for this project lacks clarity and is inaccurate and incomplete in many cases. Project’s products and services typically communicate information inaccurately. Many of the important findings or information from the knowledge base in the content area is omitted. The products and services provided usually do not communicate relevant findings (or information from the knowledge base) effectively, in clear and accessible language. In the rater’s judgment, the information conveyed would not be easily understood by the intended audience to support the use and implementation of the content in most cases.
1 “Very Low Quality”	Project materials indicate that the information conveyed in products, assistance, and advice for this project totally lacks clarity and is inaccurate and incomplete. Project products and services communicate information inaccurately. All important findings or information from the knowledge base in the content area is omitted. The products and services fail to communicate relevant findings (or information from the knowledge base) effectively, in clear and accessible language. In the rater’s judgment, the information conveyed would not be understood by the intended audience to support the use and implementation of the content.
0 “Not enough evidence to judge”	The project materials submitted for review do not contain enough information to determine a rating on the delivery of products and assistance.

Exhibit 3

Expert Panel Review

Rating Form

Technical Quality Dimension 1	Score (0-5)
Demonstrated use of the appropriate documented knowledge base: Materials for this project indicate that an accurate, comprehensive, and balanced review was conducted of the applicable knowledge base (empirical research, literature review, laws and regulations, professional wisdom) for the particular content area. Citations are provided and sources noted for the knowledge base. Research is derived from the latest generation of findings, based on relevant interventions, and from studies with appropriate designs. Legal or regulatory background informing the project’s products and services is accurately interpreted and applicable for the content area. Materials clearly note where professional wisdom or other non-empirical evidence forms the basis for findings or recommendations. Limitations in the knowledge base for the content area are adequately identified and acknowledged in products and services provided.
Notes/Explanation of Rating:
Technical Quality Dimension 2	Score (0-5)
Fidelity of Application of the Knowledge Base in Products and Assistance Provided: Materials for this project indicate that products, assistance, and advice provided to clients are consistent with the core findings from the available knowledge base. There is clear and consistent treatment in the products, advice and assistance provided in relation to the level of rigor and certainty in the available knowledge base. Preference is given to providing products, assistance, and advice based on scientifically-based research and clear legal and regulatory guidance. When only a weak research or knowledge base is available, products, assistance, and advice provided are appropriately tempered and professional wisdom is incorporated and identified in suggesting a range of viable alternatives for consideration by clients.
Notes/Explanation of Rating:
Technical Quality Dimension 3	Score (0-5)
Clear and effective delivery of products and assistance: Materials for this project indicate that products, assistance, and advice provided to clients are well-written and well-organized—clear, accurate, applicable and complete—for the intended audience. The presentation of findings, information, recommendations or assistance as contained in materials is appropriately designed for the intended audience and supports the use and implementation of the content.
Notes/Explanation of Rating:

2