Att_LRC Supporting Statement 11.5.09

The American Institutes for Research (AIR), under contract with the U.S. Department of Education, International Education Programs Service (IEPS), requests clearance for data collection to conduct an assessment of the Language Resource Centers (LRC) Program. The study will survey Program Directors at LRCs and key constituents of LRC products to gather information on the implementation of the LRC Program and inform program improvement.

This document contains three sections. The first section is a description of the assessment of the Language Resource Centers Program and provides context on the data collection instrument for which we are seeking clearance. The second section contains Parts A and B of the supporting statement for the Paperwork Reduction Act Submission. The appendices contain the instruments for which we are requesting clearance, along with additional supporting documentation.

Background

In establishing the Language Resource Centers (LRC) Program as part of the reauthorization of the Title VI Higher Education Act in 1986, Congress acknowledged the critical need to strengthen foreign language instruction in the United States. As with the original authorization of the Act in 1958—which was in response to world events that challenged assumptions about the preparedness of the United States to compete with other nations—the LRC Program was designed to ensure that American students gained the skills necessary to communicate across borders. Twenty-two years after the authorization of the Program, the need to train students in foreign languages in order to compete in the global marketplace is more essential than ever. In a world in which the military, economic, and environmental security of the United States are closely intertwined with other nations, it is imperative that American students are proficient in the multitude of languages spoken around the globe.

Since the first LRC competition in 1990, the Program has made great strides in bolstering the foreign language capacity of the United States. As noted in the recent report from the National Academy of Sciences (NAS) reviewing the Title VI and Fulbright-Hays programs, instruction in less commonly taught languages has increased and numerous teaching materials have been developed for use in a variety of educational settings.¹ The NAS noted, however, that the 14 Title VI programs, including the Language Resource Centers, had yet to comprehensively assess the degree to which the projects funded under the statute had demonstrated an effect on international education and foreign language instruction in the United States. Consequently, the level of preparedness of U.S. students to compete globally remains unclear.

Study Objectives

The primary objectives of the study are to a) describe the activities conducted with LRC funding since inception, b) examine the individual project outcomes associated with each of the Centers, and c) assess the contributions of the LRC program to the implementation of the study of modern foreign languages in the United States.

Data Collection and Instrumentation

The Assessment of the Language Resource Centers Program will use the following data collection strategies:

Review of Secondary Data. AIR will collect both qualitative and quantitative data from LRCs in order to document the range of activities and products they create and disseminate. These data will be in the form of interim and annual reports and extant evaluations conducted by the LRCs.

Survey of LRC Project Directors. AIR will survey all 16 LRC Project Directors in order to collect additional information about activities and products created by the Centers not captured during the review of secondary data. Moreover, program administrators will be queried about the evaluations Centers have conducted of their activities and the performance measures they use as part of their evaluations.
Survey of LRC Constituents. AIR will survey the key constituents of LRCs in order to gather information on 1) constituent awareness of the products and services disseminated by LRCs, 2) the frequency with which constituents use these products and services, and 3) constituent assessments of their utility. Two groups of key constituents will be surveyed: district supervisors who belong to the National Association of District Supervisors of Foreign Languages (NADSFL), and participants at LRC Summer workshops that focus on the postsecondary level. While many of the questions will overlap, two separate survey instruments will be developed for each constituent group. The purpose of surveying these two separate groups is to gather data on LRC effectiveness at both the K-12 and postsecondary levels. Focusing on one without the other would miss a critical component of the LRCs’ mandate.

Exhibit 1 provides a schematic representation of the proposed study design to gather information on the LRC program. As noted in the Exhibit, AIR will prepare a final report at the conclusion of the data collection and analysis tasks. The results from the study will be used to learn what is being accomplished by the LRC Program, whether its products are being used by its core constituents, and to inform program improvement in the future.

Exhibit 1. Study Overview

Group 41

Survey of LRC Project Directors

AIR has worked together with IEPS staff, the Technical Advisory Group (TAG) and our expert consultant to develop a survey instrument to be administered to Program Directors at each of the 16 Language Resource Centers. The focal areas covered in the survey will be 1) the staffing, funding, and management of the Centers, 2) the products and services created by a Center since receiving Title VI funding, 3) a description of the performance measures used by Centers to evaluate their program, 4) the results of any evaluations conducted by the Centers, and 5) an assessment of the successes and challenges facing the Centers and the LRC Program as a whole. The survey will combine both open-ended and close-ended questions in order to allow administrators the opportunity to elaborate upon their responses.

In addition to considering the content during survey development, it is also important to consider the length of the questionnaire. The goal of collecting comprehensive information from Program Directors must be balanced against the burden of completing a long questionnaire. Through the use of skip patterns and other techniques of maximizing efficiency, we have created a survey that minimizes respondent burden while still gathering valuable data. We expect this survey to take no longer than 90 minutes to administer.

The survey of LRC Project Directors will be disseminated via email and can be filled out directly on the computer via Word. In the event that respondents do not wish to complete the surveys electronically, paper copies will be provided. Because a 100 percent response rate from the 16 current and former LRCs is essential in documenting the activities of the Centers, project staff will follow-up with administrators until each survey is completed.

Survey of Constituents

The purpose of the survey of constituents is to gather information on the visibility and use of the products developed by the LRCs. Specifically, the focus of the survey will be on the following topics:

Constituent awareness of LRC products and/or services
Frequency of use of LRC products and/or services
Assessment of the utility of the LRC products and/or services

Although information about the reach of the Centers will be gathered through the review of secondary data and the survey of LRCs, the data collected during these phases is limited because it is self-reported by the LRCs. In order to fully measure the reach of the activities conducted by the Centers, the assessment must look beyond the LRCs to the constituents they serve. The survey of constituents will provide data on the degree to which the materials created by LRCs reach their intended audience.

We will administer two separate versions of the constituent survey: one to district supervisors who belong to the National Association of District Supervisors of Foreign Languages (NADSFL), and another to participants at LRC Summer workshops that focus on the postsecondary level. These two groups have been selected by the TAG has the key users of LRC products and services. While many of the questions will overlap, two separate survey instruments will be developed for each constituent group. The purpose of surveying these two separate groups is to gather data on the use of LRC products at both the K-12 and postsecondary levels. Focusing on one without the other would miss a critical component of the LRC’s mandate.

Constituent Survey for NADSFL Members

NADSFL consists of approximately 300 members who are foreign language supervisors across a range of school districts in the United States. They are a key target audience for LRC in that they are the folks who make decision about which products to use for teaching foreign languages in their district. AIR plans to administer the constituent survey to all individuals who are members of NADSFL during the 2009 calendar year. The survey will be administered via the web using Survey Gizmo, with an option to fill it out via paper and pencil. To minimize respondent burden while maximizing the response rate, we anticipate the survey of constituents to take no longer than 30 minutes to complete.

Constituent Survey for Postsecondary Summer Workshop Participants

While the survey of NADSFL members is essential in gathering data on the reach and use of LRC products, it is nevertheless limited because members of NADSFL focus only on the K-12 level of instruction. Many of the LRC products are targeted to the postsecondary level and to gather data on those products, we must reach constituents who work in postsecondary institutions. This is the purpose of this second constituent survey.

Specifically, as part of this survey we will gather data from the universe of postsecondary Summer workshop attendees for the year 2009. As with the NADSFL survey, the questions will focus on the participants’ awareness and use of the LRCs products (beyond those from the workshop). We anticipate that approximately 200 participants will be surveyed via the web using Survey Gizmo, with an option to fill it out via paper and pencil. To minimize respondent burden while maximizing the response rate, we anticipate the survey of constituents to take no longer than 30 minutes to complete.

Supporting Statement for Paperwork Reduction Act

A. Justification

1. Circumstances making collection of information necessary

Since the first LRC competition in 1990, the Program has made great strides in bolstering the foreign language capacity of the United States. As noted in the recent report from the National Academy of Sciences (NAS) reviewing the Title VI and Fulbright-Hays programs, instruction in less commonly taught languages has increased and numerous teaching materials have been developed for use in a variety of educational settings.² The NAS noted, however, that the 14 Title VI programs, including the Language Resource Centers, had yet to comprehensively assess the degree to which the projects funded under the statute had demonstrated an effect on international education and foreign language instruction in the United States. Consequently, the level of preparedness of U.S. students to compete globally remains unclear.

2. Purposes and uses of the data

The American Institutes for Research (AIR) is conducting an assessment of the Language Resource Centers Program. The data will be used to document the implementation of individual projects as well as of the program collectively and to inform future studies looking at long-term impact. The results from the study can be used to learn what is being accomplished by the LRC Program and to inform program improvement in the future.

3. Use of technology to reduce burden

The two constituent surveys will be administered via the web through Survey Gizmo. The use of multiple skip patterns seamlessly integrated through the on-line survey will greatly reduce burden on respondents by only presenting them with the questions relevant to them. The project director survey will be disseminated via email and can be filled out directly on a computer in Word (given the small number of respondents, the cost of developing a web version of this survey does not make sense). All respondents will have the option to request the survey in hard copy format. The survey website for the web-based surveys will be password protected.

4. Efforts to identify duplication

The assessment of the Language Resource Centers Program represents IEPS’s only study currently underway that is aimed at compiling data on each LRC program to promote foreign language education. IEPS has not conducted any program assessment since the start of the program. Thus no duplication will result as part of the study.

5. Methods to minimize burden on small entities

No small businesses or entities will be involved as respondents.

6. Consequences of not collecting the data

The Language Resource Centers Program is a government funded program. Since it has not been evaluated previously, the IEPS does not know how well it operates. The consequences of not collecting the data will result in having no information on the implementation of the program. Without the data collected as part of this study, IEPS will be unable to identify steps for ensuring that the LRC program meets the foreign language needs of the United States.

7. Special circumstances

No special circumstances apply to this study.

8. Adherence to 5CFR 1320.8 guidelines and consultation outside the agency

This study was listed in the Federal Register.

To assist with the development of the study of the Language Resource Center Program and the drafting and vetting of the surveys, project staff has drawn on the input of several outside experts who are world-renowned in the fields of foreign languages and area studies.

The members for the Technical Advisory Group (TAG) are as follows:

Nina Garrett – Former Director of the Yale University Center for Language Study
Paul Sandrock – Assistant Director for the Content and Learning Team at the Wisconsin Department of Public Instruction
Richard Tucker – Professor in Applied Linguistics in the Department of Modern Languages at Carnegie Mellon University

The consultant is as follows:

Scott McGinnis – expert in foreign language instruction and pedagogy and currently Academic Advisor and Professor at the Defense Language Institute.

Project staff will use outside experts for consultation on an as-needed basis throughout the study.

9. Payment or gifts

No payments or gifts will be used over the course of this study.

10. Assurances of confidentiality

AIR research staff will be trained to keep all names and any other identifying information completely confidential, and to omit this information while recording information from the surveys. Caution will be exercised in limiting data access to authorized project staff and those who have been instructed in the confidentiality requirements of the study. The data will contain no information that could be used to identify subjects other than that which is publicly available. No individual identifying information will appear in any of our reports. All materials will be stripped of all individually identifiable information to further protect respondent confidentiality.

11. Justification of sensitive questions

No questions of a sensitive nature will be included in this study. Respondents are reporting on program-level activities only.

12. Estimates of hour burden

The total estimated hour burden for the entire study is 236.5 hours (or 14190 minutes). The hourly burden breakdown is as follows:

LRC Project Director survey: 16 respondents * 90 minutes per survey * 100% response rate = 1440 minutes.
NADSFL constituents survey: 300 respondents * 30 minutes per survey * 85% response rate = 7650 minutes.
Summer workshop participants survey: 200 respondents * 30 minutes per survey * 85% response rate = 5100 minutes.
Total burden = 1440+7650+5100=14190 minutes=236.5 hours.

13. Estimate of cost burden to respondents

There are no additional respondent costs associated with this data collection other than the hour burden accounted for in Item 12.

14. Estimate of annual cost to the federal government

The estimated cost for the 18 month study, including development of data collection instruments, justification package, data collection, data analysis, and preparation of a final report, is $274,582 per year.

15. Program changes or adjustments

The request is for a new data collection. There are no program changes or adjustments.

16. Plans for tabulation and publication of results

AIR will submit a final report summarizing the results of the entire study. In drafting the report, AIR will first create an outline of the contents. The contents will include, but are not limited to, the following:

Executive summary
Purpose of project
Research methodology
1. Survey design
2. Data collection procedures
3. Database design
4. Data analysis
Results from the survey of LRC Project Directors and the survey of LRC constituents.
Implications of study and future directions

AIR will write one substantive report, including results for the entire project based on the original analysis plan and any subsequent agreed-upon modifications to the plan. The report will also include a description of the methodology employed, findings and implications. The report will include an executive summary in non-technical language, which will be appropriate for a wide range of audiences. In addition, AIR will prepare three two-page executive summaries that report on key topics of interest to IEPS. The final report and summaries will be provided in three formats: camera-ready copy, Microsoft Word, and a copy compatible with IEPS’s Website formatting.

17. Approval to not display OMB expiration date

All data collection instruments will include the OMB expiration date.

18. Explanation of exceptions

No exceptions are requested.

B. Description of Statistical Methods

1. Universe and Respondent Selection

All three surveys proposed in this study are targeting the entire universe of respondents (i.e., census). This is because the number of cases is small, and sampling would not be appropriate.

The frame for LRC Project Directors is derived from the U.S. Department of Education’s International Resource Information System (IRIS) and includes all 16 grantees who received LRC funding since program inception. The frame for the NADSFL survey will be obtained from the Executive Director of that organization and will contain the full list of 2009 members (approximately 300 members). The frame for Summer Workshop participants will be obtained from the LRC Project Directors based on their records of who attended the 2009 Summer workshops and was employed by or enrolled in an institution of higher education (approximately 200 participants across all 16 LRCs).

These frames are as exhaustive as frames can be in that they represent the entire population of interest. Because membership in NADSFL and enrollment in Summer Workshop is voluntary, it is possible that the universe of participants is not similar to the larger population of users of LRC products across the United States. However, because there is no such frame and the population of LRC product users is undefinable, this limitation will be noted in the final report.

2. Procedures for Collecting Information

The information for this study will be gathered through three surveys (see Appendix A). Two of these surveys (NADSFL and Summer Workshop participants) will be administered via the World Wide Web using Survey Gizmo, a commercial survey dissemination and management website. Because of the small number of respondents (N=16), the LRC Project Director survey will be administered as an electronic Word document disseminated via email. All respondents will be able to request a pencil and paper version of the study.

Respondents to all three surveys will be contacted about the study via email. AIR has already worked with LRC Project Directors and the NADSFL Executive Director to publicize the study and emphasize its importance to respondents. All respondents will receive a prenotification email about the study emphasizing its importance and requesting their participation. Three to five email follow-ups over a period of two months (depending on response rates monitored on a weekly basis) will follow up with respondents to encourage participation (see Appendix B for contact and follow-up emails).

3. Methods to Maximize Response Rates

As discussed above, methods to maximize response rates have already been implemented by publicizing the study early on to respondents via email to Project Directors and members of NADSFL. In addition, the follow-up procedures discussed above will be implemented to follow-up with respondents who have not filled out the surveys.

Because the study has been well publicized already and the respondents are professionals with a vested interest in the topic of the study and in the future of the LRC program as disseminators of foreign language resources, we expect a high response rate.

It is expected that the response rate for the LRC Project Directors will be 100 percent because they are all currently grantees of the U.S. Department of Education with a vested interest in the study.

The expected response rates for the NADSFL members’ survey and the Summer workshop attendees is 85 percent. This response rate will be achieved through a combination of contact letters emphasizing the importance of the project and endorsed by influential scholars in the field of foreign language teaching, and the multiple follow-ups from the contractor mentioned above.

Non-response bias due to a less than 100% response rate will be examined by comparing the background demographic information available on the frame (e.g., gender, institutional affiliation, region of the country) to that of the respondents. While no statistical adjustments are planned given we are using a census, an analysis reporting on the frame totals as compared to the respondent totals will be reported in the final report and appropriate caution in the interpretation of findings will be discussed.

4. Tests of Procedures

Because the questions in these surveys capture implementation and are process-oriented, they have not been cognitively tested. They have, however, been vetted by a committee of experts, and were developed, in part, based on a review of similar questionnaires used previously by the U.S. Department of Education.

Usability of the website for the surveys will be examined by AIR’s subcontractor Firepig Partners who specializes in on-line survey design and administration. Based on their recommendations, revisions to the web designs will be implemented.

5. Contacts for Statistical Aspects and Data Collection

The following individuals were involved in the design and statistical aspects of the study and its data collection:

Stephane Baldi, Principal Research Scientist, American Institutes for Research
Tanya Taylor, Research Analyst, American Institutes for Research

Nina Garrett – Former Director of the Yale University Center for Language Study
Paul Sandrock – Assistant Director for the Content and Learning Team at the Wisconsin Department of Public Instruction
Richard Tucker – Professor in Applied Linguistics in the Department of Modern Languages at Carnegie Mellon University

Scott McGinnis – Academic Advisor and Professor at the Defense Language Institute.

1 National Research Council (2007). International Education and Foreign Languages: Keys to Securing America’s Future, Committee to Review the Title VI and Fulbright-Hays International Education Programs, M.E. O’Connell and J.L. Norwood, Editors. Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

2 National Research Council (2007). International Education and Foreign Languages: Keys to Securing America’s Future, Committee to Review the Title VI and Fulbright-Hays International Education Programs, M.E. O’Connell and J.L. Norwood, Editors. Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	INTRODUCTION AND OVERVIEW
Author	American Institutes for Research
File Modified	0000-00-00
File Created	2021-02-03