Response to OMB Comments

Responses to OMB Comments on QRIS System Clearance Package

Question 1: Please provide a little more information on how studies identified for inclusion in FRSS or PEQIS. For example, does NCES pro-actively talk with program offices about the “service” that QRIS provides? Are there folks in IES, RIMS, or elsewhere who routinely ‘direct’ folks towards QRIS as a good option for certain kinds of studies? Does NCES believe that QRIS is well known across program areas?

Question 2: Somewhat on the flip side, does NCES ever experience back-ups in the queue? How generally is timing and assurances for various clients managed?

Response: The following response addresses both questions 1 and 2. NCES has established two survey systems (FRSS and PEQIS) to collect time-sensitive, issue-oriented data quickly with minimal response burden. FRSS began in 1975 and until 1990 collected data at all educational levels. In 1991, NCES established PEQIS for the purpose of collecting data from postsecondary institutions. Taken together NCES uses these systems to meet the data needs of Department of Education analysts, planners, and decision makers when information cannot be obtained quickly through traditional NCES surveys.

NCES believes that QRIS (FRSS/PEQIS) is well known within the Center and across the various Department of Education program areas. In fact, we regularly collaborate with and/or conduct studies on behalf of many Department of Education offices. This includes the Office of Educational Technology (OET), the Office of Innovation and Improvement (OII), the Office of Safe and Drug-Free Schools (OSDFS), the Office of Elementary and Secondary Education (OESE), the Office of Postsecondary Education (OPE), and the Office of Special Education and Rehabilitative Services (OSERS).

QRIS studies come about in one of two primary ways. One is that a Department of Education office directly contacts QRIS staff or senior-level NCES management about requesting a study on their behalf. The second is that NCES internally initiates a QRIS study. Regardless of how a QRIS study is initiated, appropriate Department of Education program office staff or NCES staff with substantive expertise in the area a particular survey focuses on are actively sought out for consultation. A recent example of this includes a NCES initiated PEQIS survey on the institutional perspective of students with disabilities. Since a PEQIS survey on this topic was last conducted in 1998, QRIS staff contacted staff from OSERS (who also collaborated on the 1998 survey effort) and the IES, National Center for Special Education Research for consultation. Staff from both offices reviewed the 1998 draft and offered comments on how we should update the instrument to more accurately capture current issues, with OSERS staff also commenting on subsequent drafts that emerged during survey development.

The typical QRIS workload usually ranges between four to six ongoing surveys in various stages of operation (e.g., survey development; data collection; post-data collection analysis and report preparation). Under these conditions it is not an issue to begin discussions within NCES or with other Department of Education program offices about undertaking a new survey study right away. However, the current QRIS workload includes an unusually high number of surveys; specifically 15 ongoing surveys. Four of these studies are currently either wrapping up data collection activities or conducting post-data collection analysis and report preparation. The remaining 11 surveys have data collection activities scheduled at various times beginning this summer through the end of the 2009-10 school year. With this current workload we’ve had to set timeframes for when we can take on new survey work; more importantly understanding our timelines for when we can realistically begin actual data collection activities. To that end, in recent weeks we’ve had conversations with a couple potential clients about surveys they are interested in. In both cases we informed them that the current QRIS workload restricts our capacity to begin new work immediately and that January 2010 is the earliest we could begin survey development work in time for a data collection beginning no sooner than the summer of 2010. In one case, our timeframes did not fit the potential client’s preferences for a data collection during the 2009-10 school year. In the other case, the potential client indicated that the scope of the survey topic was still being defined, but that the QRIS timeframes posed no problem for them. Should this client approach us in the coming weeks or months about moving forward with a survey, we will slot them first in the queue to begin survey development in January 2010.

With that being said, QRIS staff regularly review the current workload and project out the expected timelines for when specific surveys will be completed (i.e., through publication of results and receipt of data files). This allows QRIS staff to communicate with potential clients the timelines for incorporating their request into the rotation of survey work being completed.

Question 3: Does NCES impose any limits in its sampling to ensure that the same school or district is not sampled greater than a certain number of times in a year or other time period? What about for postsecondary institutions?

Response: QRIS does not usually conduct multiple surveys of a particular education sector (schools, districts, or postsecondary institutions) in a time frame that would impose undue burden, or the perception of undue burden, on entities in that sector. However, when there are instances in which burden for a particular sector may be higher than usual due to multiple surveys of a particular sector, NCES handles this by minimizing sample overlap. For schools and districts, this is done by minimizing (but not necessarily eliminating) overlap during sample selection through the use of conditional probabilities of selection that take into account the joint selection probabilities for each survey. For example, the paper by Chowdhury, Chu, and Kaufman (2000), “Minimizing overlap in NCES surveys,” Proceedings of the Survey Methods Research Section, American Statistical Association, pp. 174-179, describes a general approach that can be used to minimize sample overlap among several existing surveys while at the same time achieving the desired overall probabilities of selection for the planned survey. This procedure was used to select the school sample for FRSS 67 (arts education) in 1999, when NCES requested that the school samples for FRSS, NAEP, and SASS be selected to minimize overlap among the three concurrent surveys. For postsecondary institutions, the 1,627 institutions in the PEQIS panel would be split into three sub-panels, and each survey would be sent to two of the three sub-panels on a rotating basis (A & B, B & C, A & C). This procedure provides a sample size of almost 1,100 institutions per survey, while reducing overall burden on the institutions. This procedure of using two of the three sub-panels was used for many of the early PEQIS surveys, including PEQIS 1 through 4, PEQIS 6, and PEQIS 8. More recent PEQIS surveys have not used the sub-panels since the surveys were not conducted close together in time.

Question 4: Are the postsecondary institution contacts the same as for IPEDS or not?

Response: At the time the PEQIS panel was established, the president of each sampled institution designated an institutional coordinator. These coordinators facilitate data collection by identifying the appropriate respondent within the institution for each PEQIS survey (depending on survey topic) and ensuring that the questionnaire reaches the respondent. The suggested coordinator was someone in the institutional research office, but the choice of the coordinator was left to the discretion of the president. Since the roles of the PEQIS and IPEDS institutional coordinators are very different, no attempt was made to have the coordinators for the two systems be the same person.

Question 5: Please describe the coordination between QRIS and the IPEDS staff on all aspects of the program, such as identification of coordinators, topics, timing, etc.

Response: The coordination between QRIS and the IPEDS staff revolves around ensuring that PEQIS surveys avoid duplication of data being collected regularly through IPEDS. This is because PEQIS survey topics can vary widely in response to either NCES initiated studies or from requests by other Department of Education offices. In short, upon receipt of a PEQIS survey request, QRIS staff contact an IPEDS representative to confirm whether a topic has been addressed and, if so, how.

More broadly, to avoid duplication of effort across all FRSS and PEQIS surveys QRIS staff initiate communication with the NCES project officers who oversee data collections that might address the topic being requested. Moreover, a NCES Questionnaire Review Board (QRB) is formed to review each FRSS and PEQIS survey just prior to submission for OMB clearance. The QRB is usually composed of several NCES staff with expertise in areas that is closely aligned to the FRSS or PEQIS survey topic.

Question 6: Does IPEDS have a representative sample of less-than-two-year schools that would be adequate for PEQIS?

Response: Yes. IPEDS includes all Title IV eligible postsecondary institutions, including 4-year or higher (referred to by IPEDS as 4-year institutions), at least 2 but less than 4-years (referred to as 2-year institutions), and less-than-2-year (below associate’s) institutions. All institutions, including less-than-2-year institutions, that have Program Participation Agreements (PPAs) with the U.S. Department of Education (indicating institutions that are eligible to award Title IV federal financial aid) are required to complete IPEDS, and therefore are represented on the IPEDS frame.

Question 7: Please clarify the overall mission of NCES with regard to public libraries since 2007, when responsibility for some collections moved to IMLS, including the survey that QRIS proposes for a frame.

a. Please clarify how that change does or should affect QRIS’s scope and frame plans for the future.

Response: The file that QRIS would use as a sampling frame for library samples is the public library universe file. This file was originally developed by NCES, but is now maintained and updated annually by IMLS. The following statement is from the IMLS website:

“Universe file - NCES also developed the first comprehensive public library universe file. Updated annually, it is part of WebPLUS and includes identifying information on all known public libraries and their service outlets. This resource is available for drawing samples for special surveys. IMLS will continue to maintain this universe.” If a library sample is needed for QRIS, the public library universe file will be obtained from IMLS rather than NCES.

The Education Sciences Reform Act of 2002 (HR3801) Part C Sec 151(b)(3)(B) states that the mission of NCES shall be to collect, analyze, and report education information and statistics in a manner that is relevant and useful to practitioners, researchers, policymakers, and the public. Sec 153(a)(1)(O) of the Act specifically authorizes NCES to collect, acquire, compile, and disseminating full and complete statistics on the condition and progress of education, including data on the existence and use of school libraries. Sec 154(b)(2)(A) of the ACT further states that NCES Commissioner may, as appropriate, use information collected from libraries. Lastly, Sec157 of the Act indicates that NCES may establish 1 or more national cooperative education statistics systems for the purpose of producing and maintaining comparable and uniform information and data on libraries, that are useful for policymaking at the Federal, State, and local levels. In 2007, NCES passed over to IMLS one of its data collections pertaining to libraries. This collection defines the universe of all public libraries in the United States. Until the re-authorization of the 2002 Education Sciences Reform Act, NCES retains the authority to collect data on public libraries, such as through the QRIS surveys.

Question 8: Please clarify the number of people that are contacted during a typical “pretest with nine institutions.” OMB wants to be sure that NCES can perform adequate pretesting, but such pretesting must be submitted through NCES’s formative generic ICR if contacting more than 9 individual persons.

Response: A typical “pretest with nine institutions” involves receiving written feedback (in the form of a completed questionnaire) and verbal feedback (in the form of a followup telephone interview) from 9 individual respondents who provide information for their institution (typically a school, district, or postsecondary institution). For example, a typical pretest of a school-level survey involves collecting information (questionnaire and telephone interview) from 9 or fewer school principals. Respondents collect information for the pretest within their institution in the same ways they collect information for the full survey, including checking paper or computerized records, running tabulations, and checking with colleagues. However, contact for the QRIS pretest is limited to the 9 individuals who are providing information on behalf of their institution.

Question 9: Does FRSS use the same vintage of PSS and CCD for all three years of the clearance or is it updated as new information is available?

Response: FRSS uses the most recent versions of PSS and CCD that are available at the time that a particular sample is drawn. The version being used is specified in the sampling section of the clearance materials submitted to OMB for each FRSS survey.

4