Appendix A - LTO Outreach Cover Memo

Appendix A 

LTO Outreach Informant Interviews Cover Memo

To: [Study Stakeholder]

CC:

From: LTO Project Team (MDRC)

Date:

Subject:

1. Project Summary and Request

The goal of the Long-term Outcomes Project (LTO) is to learn more about the current parameters for linking evaluation and administrative datasets to examine long-term outcomes and impacts of government-funded programs and interventions. The Administration for Children & Families’ Office of Planning, Research & Evaluation (ACF/OPRE) is the federal sponsor for this project, and MDRC is the contractor.

 

In the first stage of the project, evaluations and administrative data sources were selected and reviewed with the goal of assessing the practical, legal, and ethical feasibilities for accomplishing the linkages. We are conducting interviews in this second stage of the project to learn more about the steps and resources that may be required to conduct long-term follow-up research for the [insert evaluation].

If Wave 1 or talking to Wave 1 respondents for Wave 2:

[During the first stage of the project, we reached out to you because the evaluation you worked on, [insert evaluation here], was one of the studies we reviewed. We sent you a data collection template requesting that you review the information and fill in any missing details. The completed template you returned to us is attached for your reference).]

If Wave 2, new respondents:

[During this first stage we reached out to [insert lead informant name] to learn more about the [insert evaluation here]. We sent [lead informant] a data collection template requesting that s/he review the information and fill in any missing details.]

We are reaching out to you today because, [insert abbreviated study name here], is one of 8 studies we are still reviewing, and we are connecting with additional study stakeholders to gather more information that could inform plans for conducting long-term follow-up research for this study. This information will be used to prepare a long-term outcome feasibility report for each study in our sample.

If Wave 1:

[We are following up with lead study informants to reconfirm some details from the earlier round of data collection. Another key purpose of this interview is to identify additional individuals we could speak to about the availability and quality of the evaluation data, the data sharing agreements/contracts, and potential administrative data sources for long-term follow-up research for this evaluation.]

If Wave 2:

[We recently connected with [insert Wave 1 informant] to about [the evaluation], and s/he recommended that we connect with you for more information. ]

We’d like to speak with you about [insert topic depending on informant type here]. The interview will take no more than 60 minutes and includes time for you to ask us any questions afterwards. We would like to conduct the interview by [insert date here]. Would you like to be interviewed?

Below is a FAQ about the project. If you have any questions or would like to schedule a brief call to talk about the project prior to our data collection interview, please reach out to Alexandra Pennington at 212-340-8847 or Alexandra.Pennington@mdrc.org and/or LTO@mdrc.org.

2. FAQ

1. Who is the Project Director?

Alexandra Pennington, Research Associate

2. Who is the ACF/OPRE Project Officer for the project?

Brett Brown, PhD, Senior Social Science Research Analyst

3. Is MDRC contracted to conduct long-term follow-up for this project?

No, the goal of this project is to understand what steps and resources may be required to conduct long-term follow-up for this study and other similar studies. No long-term follow-up analysis is being conducted for this project.

4. What are ACF/OPRE’s goals for this project?

ACF/OPRE wants to better understand the current parameters for and the steps that may be required for linking employment- and child/youth development-related evaluations to administrative datasets (for long-term follow-up research purposes). MDRC is preparing an internal report for ACF/OPRE that will summarize these parameters and steps, by study.

5. How did the project team determine that this study (and other similar studies) is “potentially feasible” for long-term follow-up?

During the first part of the project, MDRC took a four-phase approach to study selection. In Phase 1: Scan, we reviewed evaluation reports to see if they met ACF/OPRE’s basic criteria for assessment. In Phase 2: Curate, we narrowed this list down to 16 to 25 “major evaluations” based on the rigor and content of the studies, while capitalizing on information holdings at existing clearinghouses. In Phase 3: Collect, we collected more in-depth information on major evaluations and the potential for matching to administrative records. This included information on data ownership, existence of personally identifying information, IRB and consent form allowance and restrictions, and past findings. In Phase 4: Analyze, we analyzed the full range of feasibility considerations for those studies and potential approaches for overcoming any challenges. We concluded with giving a feasibility rating for the final list of 25 evaluations. The project team ultimately identified 13 evaluations as potentially feasible candidates for long-term follow-up based primarily on the existence of the study participant personally identifying information (PII) needed for matching and key outcome data from an administrative data source. More information can be found on these slides, presented at the Association of Public Data Users Conference in 2019.

6. What are some topics or sample questions that I may be asked?

   1. For project directors/principal investigators

      1. What research questions from the original evaluation, if any, would long-term follow-up help to address?

      2. Can you identify stakeholders to talk to with knowledge of the above-mentioned areas?

   2. For managers of evaluation data

      1. What types of personally identifiable information are available?

      2. Can you describe the types and sources of study participant outcome data (e.g. follow-up survey, administrative data, etc.) available?

   3. For evaluation legal/contract representatives

      1. Which organizations or agencies are listed as parties to the agreement?

      2. If a contract renewal were needed to facilitate long-term follow-up, can you describe any apparent factors that may help or hinder this process?

   4. For evaluation IRB representatives

      1. Did the informed consent form indicate how long study participant’s information would be collected? If so, for how long?

      2. How much additional follow-up, beyond the follow-up that’s already been done, is covered by the informed consent?

   5. For representatives for study oversight (e.g. funder)

      1. What is the process for getting a contract in place to do long-term follow-up?

      2. Who would be permitted to be a party to this contract (original evaluator, others?)?

   6. For administrative data source providers

      1. What is the process for getting a data sharing agreement in place for your data source?

      2. What are the associated costs? Can you give us an example?

   7. For other kinds of stakeholders

      1. Are there any additional research questions, beyond what were in-scope of the original evaluation, that might be interesting to include in the long-term follow-up?

      2. How do you think the original research or long-term follow-up effort can be made more accessible to the population it would ideally impact or support?

7. What documents should I review in preparation for the interview?

   1. For project directors or principal investigators

      1. Study contracts, study findings, data sharing agreements, informed consent form, IRB application, data dictionaries and codebooks, etc.

   2. For managers of evaluation data

      1. Data sharing agreements, data dictionaries and codebooks, informed consent forms, etc.

   3. For evaluation legal/contract representatives

      1. Informed consent forms, data sharing agreements, memorandums of understanding, etc.

   4. For evaluation IRB representatives

      1. IRB application, data sharing agreements, informed consents forms, etc.

   5. For representatives for study oversight

      1. Study contracts, etc.

   6. For administrative data source providers

      1. Data sharing agreement or application

   7. For other kinds of stakeholders

      1. Study findings

8. How long will interview preparation take?

We expect interview preparation to take no more than an 90 minutes. This includes the gathering and review of the documentation detailed in Question 7.

9. Will my decision to participate or not participate in this interview preclude my organization from receiving funding to conduct long-term follow-up for this study?

No; if ACF/OPRE later decides to provide funding for long-term follow-up for the study, this funding opportunity may be released as part of a competitive bidding process. Study stakeholders are encouraged to participate to help ACF/OPRE better understand the process and associated resources needed to pursue a long-term follow-up effort for this study.

According to the Paperwork Reduction Act, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for the described information collection is 0970-0356 and the expiration date is 6/30/2021. 

4