YEARS_Memo for revisions to program observation protocol

MEMO

To: Steph Tathum; Office of Information and Regulatory Affairs (OIRA), Office of Management and Budget (OMB)

From: Samantha Illangasekare, Office of Planning, Research and Evaluation (OPRE), Administration for Children and Families (ACF)

Re: Non-substantive change request for revisions to program observation protocol for the YEARS information collection (OMB # 0970-0470)

Date: February 11, 2016

Based on feedback from three initial site visits, we have made a few minor updates to the Youth Education and Relationship Services (YEARS) program observation protocol (approved under OMB#0970-0470). This memo outlines each proposed revision and the corresponding rationale. The proposed changes do not increase participant burden and can be considered administrative/non-substantive changes. Tracked and clean versions of the revised observation protocol are attached.

Proposed revisions to observation protocol

General updates

• Delete the start and end time rows; replace with a row that asks the time of day the session took place.

  • Rationale: We only need to know the duration of the session and if the session began within 5 minutes of its intended start time (this information is already included). We propose adding the time of day to help assess whether the sessions occurred in school or out of school (setting is already asked).

• Delete the rows for number of male and female participants; record the total number of participants instead and whether the group was predominately male, predominately female, or about equally mixed.

  • Rationale: The exact numbers of males and females were difficult to count with larger groups. The specific gender of participants was also sometimes unclear. We do not need the exact count of males versus females for the purposes of the study.

Updates to HMRE topics covered

• Take out “Skills were practiced…” columns and make a single row asking yes/no if students practiced any skills and which specific types of skills if able to be determined.

  • Rationale: It was very difficult to determine which specific skills were practiced. Some skills practiced seemed to cover multiple categories and we believe the data would be more accurate if we record whether any skills were practiced. We will still identify the specific types of skills when possible.

• Add a column to note if the team should review the “Notes” column to assess any specific issues/concerns with how topics were addressed. For example, if topics were covered in an inappropriate way (e.g., victim blaming, promoting traditional gender roles; not responding to youth questions), observers will note these specific concerns in the “Notes” section, and will flag whether the notes section should be reviewed.

  • We already include a notes section, but we want to ensure analysts check the notes as they code data if concerns were raised during the program observations. This will give a marker for analysts to be able to correctly code the final data. Notes will also give a richer qualitative description of particularly effective delivery as well as more ineffective delivery of services.

Updates to fidelity monitoring information

• Delete row about tools used for monitoring fidelity from observation.

  • Rationale: It was difficult to tell whether a program used tools to monitor fidelity without asking them explicitly. The staff may complete a fidelity monitoring log after the observers have left and fidelity monitoring is addressed in the survey and the interviews so it is not necessary in the observation protocol.

2