ED-OMB Q&A September 2008

Strengthening Adult Reading Instructional Practices (SARIP)

200807-1830-001

Response to OMB Questions

September 23, 2008

	Response	Location in Document
1. Supporting Statement	The Supporting Statement has been edited to make it clear that the $20 payment is an incentive.	Part A, A9, page 9.
a. Please edit the Supporting Statement to reflect that it is an ED document, not a JBL document	The Supporting Statement has been edited to reflect that the Supporting Statement is an ED document.	Part A, cover, all pages except 14, 16, 18, 20, 21. Part B, cover, all pages except 9, 11, 12, 15.
b. The Supporting Statement discusses possible actions if the study finds improvements in STAR-instructed learners’ performance over non-Star instructed learners. Please clarify what types of program improvements can be made as a result of the study findings. Please also describe how a decision for a more formal study using an experimental design would be made.	The study is examining the reading instructional content, reading instructional methods, and operational characteristics of ABE programs to identify the differences between programs that have instructors that are high implementers of STAR and those that have non-STAR trained instructors. If the programs with learners that are taught by high-implementing STAR instructors make greater reading gains than learners in non-STAR programs, the characteristics of the reading instruction and the programs' operations will be examined to identify: a) the extent to which the STAR programs implemented STAR as prescribed in the training, and b) the differences between ABE programs using STAR and those not using STAR. One program change that may be indicated from the results of the study is the way in which diagnostic results are used, which is a key practice in the STAR program. For example, if there are differences in the reading outcomes of learners taught by STAR-trained instructors, where the STAR-trained instructors of higher-performing learners analyze and use the reading diagnostic instruments in greater detail than the STAR-trained instructors whose learners make less progress, then the training component of STAR may be enhanced with regard to the use of diagnostic instruments. Another type of change may relate to the type of ABE program support that is provided to reading instructors. If the learners who make the greatest reading gains are in ABE programs that provide ongoing support to reading instructors, such as providing planning time and opportunities for ongoing professional development, these practices would be emphasized in the STAR training. Other possible STAR program changes may relate to the types of reading instructional strategies that instructors used and the sequence in which reading components (alphabetics, fluency, vocabulary, and comprehension) are presented. OVAE's decision to move forward with a study using an experimental design will depend on the results from the current SARIP study and the availability of funds. If the results from the SARIP study indicate that the learners who are taught by high-implementing STAR-trained instructors perform significantly better on the reading measures used in the study than learners who are taught by non-STAR trained instructors, and if the STAR-trained instructors in the study are confirmed to have a high degree of fidelity in using STAR, then OVAE will consider moving forward with an experimental study. OVAE will also engage in an updated literature review to ascertain the extent to which an experimental design study based upon the STAR intervention will contribute to increasing or enhancing the most current evidence base around adult reading. Thus, after determining the need, fiscal feasibility, and potential return on investment of such an experimental design study, OVAE will then follow the Department’s annual spending plan process to obtain Departmental and OMB approval for the study.	Page 1 of Response to OMB Questions
c. Please describe what action the program is likely to take if the study does not show that STAR-instructed learners perform better than non-Star instructed learners.	Any action OVAE might take will need to be grounded in the results of the SARIP study. If the study does not show that STAR-instructed learners perform better than non-STAR instructed learners, than OVAE must first ascertain what the difficulties were, the scale of these difficulties and the extent to which the SARIP study’s findings support additional investment in improving the STAR intervention model. Several questions will need to be explored. For instance, OVAE would verify that the intensity and duration of the STAR instruction provided to learners during the study’s data collection period was adequate for the instrumentation used to measure learner progress. Likewise, OVAE would analyze STAR teacher implementation data collected in the SARIP study and, in conjunction with STAR developers, assess the extent to which the STAR training was implemented as intended. OVAE would also conduct analysis to determine the extent to which learners in the SARIP study were representative of learners throughout the entire Federal adult education system nationwide. Based on analyses such as these, OVAE would determine the extent to which the SARIP study’s findings support additional investment in improving the STAR intervention model. If the results of the analysis produced evidence that further investments were needed, OVAE would use the analysis to create and prioritize a list of improvements to the STAR intervention model that could be undertaken in future spending plans subject to availability of funds.	Page 1 of Response to OMB Questions
d. Please change all references to confidentiality to say “your responses are protected under the Privacy Act” or something similar that acknowledges the many permitted routine uses of data collected with the only confidentiality protection being the Privacy Act. If you are invoking another statute, please provide that information to us.	All of the references to confidentiality have been changed to say “your responses are protected under the Privacy Act.” References to “strict” confidentiality have been struck from the documents where it appeared.	Part A, page 10 Part A, Appendix J, Amended Learner Consent for Participation in Research Form, page 2 Part A, Appendix K, Introductory Letter to Program from ED Part B, Appendix A, Amended Introduction to Study, page 1. Part B, Appendix B, Amended Frequently Asked Questions from Learners Supporting Statement, Part A, page 11
e. Please clarify how students are recruited into the study? Is participation voluntary? Where are recruitment materials?	Students will be recruited for the treatment group in the SARIP study (a comparison group will be formed from existing data from two other adult reading studies). The STAR-trained reading instructors in the treatment group and the study’s local data collectors at each treatment program will recruit the study’s students. Prior to the beginning of the winter term 2009 (January), the SARIP study’s researchers will train the treatment group instructors and the local data collectors on the procedures and materials that they will use to recruit study participants (Part B, Appendix A, Amended Introduction to Study; Part B, Appendix B, Amended Frequently Asked Questions from Learners). The treatment instructors and data collectors will make a joint presentation about the study to the treatment classes during the first week of the winter term. All students in the treatment instructors’ classes will be invited to participate in the study, and they will be given an overview of the study (Part B, Appendix A, Introduction to Study). The teachers and data collectors will answer any questions that students have about their participation in the study. Participation in the study is voluntary, and students who are interested in participating in the SARIP study will meet individually with a study data collector and will be asked to sign a Learner Consent for Participation in Research Form (Part A, Appendix J). The recruitment process that will be used in the SARIP study is based on the process that the contractor (Abt Associates) used in two prior national adult reading studies. In these two studies, the contractor was able to recruit 98 percent of the students that had enrolled in the classes as study participants.	Part B, Appendix A, Amended Introduction to Study Part B, Appendix B, Amended Frequently Asked Questions from Learners
f. Please clarify when the 5 assessment instruments and background interview will be administered at program start and end? All in one sitting? Integrated with class time?	The five assessment instruments and the learner background interview will be administered during the first two weeks of the study’s classes (pre-test) in January 2009 and during the last three weeks of the classes (post-test) in May/June 2009. The comprehension and vocabulary tests, which are group-administered tests, will be administered during class time (approximately one hour). The other instruments and the background interview are individually administered instruments, and these will be administered outside of class time. The SARIP study’s local data collectors will schedule appointments with students for the assessments and interviews. All of the individually administered assessments will be conducted in one sitting along with the background interview. If students need to schedule a second appointment, the background interview will be conducted during the second appointment.	Part B, pages 4, 13
g. Regarding response A8, is this accurate? It notes that the Federal Register notices have not yet been published.	The Federal Register notices were published on May 14, 2008, Vol. 73, No. 94, pg. 27811-27812 and on July 18, 2008, Vol. 73, No. 139, pg.41346. A8 has been revised to reflect this information.	Part A, page 9
2. Student Questionnaires
a. What are the 3 to 5 most critical background variables for matching STAR-instructed learners from non-STAR instructed learners?	The most critical background variables for matching STAR-instructed learners to non-STAR instructed learners are: age, gender, place of birth and education (US/non-US), self-report of current learning problem, and pre-test score. These variables have been found to be important predictors of adult basic education students’ reading development (data from Abt Associates’ two reading studies). The Learner Background Interview Protocol, Form A and the Learner Background Interview Protocol, Form B, have been used in two adult reading studies conducted by the contractor with 1,500 learners who had reading skills at the 4th-6th grade equivalence—the same population as the learners who will participate in the SARIP. None of the learners in the two previous studies had difficulty responding to the questions on the protocols. However, for the purposes of the SARIP additional questions have been deleted from Background Interviews Form A and Form B that are not critical for the analyses in the study. OVAE considers that the remaining questions on Background Interview Form A and Form B are necessary for the study.	Part A, Appendix B, Learner Background Interview Protocol, Form A, pages 3, 5, 6, 8, 9, 12, 13, 14, 16, 17, Part A, Appendix C, Learner Background Interview Protocol, Form B, pages 7, 9, 10, 11, 12, 13,
b. Please provide a justification for the battery of questions asking about students’ other past or present adult education and vocational coursework, including dates and names of courses. Specifically, please explain how these specific elements will be used. Can these questions be trimmed or eliminated.	Previous adult reading studies conducted by the contractor (Abt Associates) have indicated that adults’ experience (current and past participation) in adult education has been related to the development of their reading skills. Thus questions in the SARIP study about students’ other past or present adult education classes are needed so that these variables can be analyzed in the outcomes analysis. However, the names of the classes and the month that adults’ participated have been eliminated. Also, the questions about students’ enrollment in vocational or work skills have be eliminated.	Part A, Appendix B, Amended Learner Background Interview Protocol, Form A. page 6.
c. Similarly, with the reading habits, activities with children, health, household, and child care sections, please explain specifically how these questions will be used. Can these questions be trimmed or eliminated?	The data on adults’ reading habits and activities with children are additional indicators of adults’ development of their reading skills and will be analyzed in the SARIP study as an outcome. The health data (e.g., vision problems, hearing problems, learning problems or disability) have been found to be related to adults’ development of their reading skills and these health variables will be analyzed as co-variates in the study. The questions on household and child care have been deleted.	Part A, Appendix B, Amended Learner Background Interview Protocol, Form A. pages18, 19. Part A, Appendix B, Amended Learner Background Interview Protocol, Form B. pages14, 15, 16.
d. Why are you asking date of birth rather than age?	The date of birth is being asked to ensure the collection of accurate data. The results from the contractor’s (Abt Associates) previous studies have shown that adult basic education respondents often cannot report their exact age when asked in an interview; thus, the SARIP study will ask students for their date of birth.
e. Can you build in a skip pattern after question 31 for those responding yes (i.e., in those cases, it is necessary to ask about parental birthplace)?	A skip pattern has been added to former question 32a (now 31a) for those answering “English” to eliminate asking respondents about their parents’ birthplaces and whether they read and spoke in English. Former Question 32a was retained since adults’ first language has been found to be one of the variables related to adults’ reading development. This variable will be examined in the SARIP study’s analysis.	Part A, Appendix B, Amended Learner Background Interview Protocol, Form A, page 16.
f. Please edit the race question to meet OMB standards. Specifically, the instruction must include a phrase such as “select one or more” and “other” is not an acceptable response to provide to respondents.	The race question has been edited to meet OMB standards.	Part A, Appendix B, Amended Learner Background Interview Protocol, Form A, Question 32b, page 17.
g. Why do you need to ask multiple SES measures, i.e., educational attainment, income, and public assistance?	The available studies involving adult basic education students have indicated that educational attainment and income are not always aligned and that it is important to obtain independent measures of these variables.
3. Letter to Program Directors
a. Why is this coming from the contractor? Shouldn’t it be from ED?	A letter from ED (Part A, Appendix K, Introductory Letter to Program from ED) has been added to the study. This letter will introduce the SARIP study to the programs with STAR-trained instructors and notify programs that the contractor will be contacting them for information. After the contractor has screened and selected programs for the study, the contractor will contact the selected programs by telephone and invite them to participate in the SARIP study. During this telephone call, the contractor will explain the activities that the programs will be expected to participate in during the study. After the program’s director has told the contractor that the program will participate in the study, the contractor will send a follow-up letter to the program director (Part A, Appendix I) that confirms the telephone conversation and describes the program’s activities in the study. The first letter will only be sent to the programs that will participate on the telephone screening for the SARIP study.	Part A, Appendix K, Introductory Letter to Program from ED Part A, Appendix I, Program Participation Letter.
b. When was the agreement made, and who are the parties to that agreement?	See response to Question 3a above.
c. Is this letter the first time the Program Directors are finding out what they have agreed to do? If not, what is the purpose of this letter?	See response to Question 3a above.
d. What is the Learner Consent Form? We don’t seem to have a copy of it; please send it over.	The Learner Consent for Participation in Research Form provides study participants with information about the study, and participants are asked to sign the agreement to indicate that they are agreeing to participate in the research. Participants are given a signed copy of the form. The form contains information about the purposes of the SARIP study, what participants will be asked to do in the study, the benefits of their participation in the research, confidentiality information, how participants can withdraw from the study, and whom to contact if participants have questions. Instructions have been added to the Learner Consent Form for the Data Collector to read the form to the participant. The form has been edited and simplified.	Part A, Appendix J, Amended Learner Consent for Participation in Research Form Part A, Appendix J, Learner Consent for Participation in Research Form, pages 1 and 2

8