ECA Language Supporting Statement Revised (9-05-12)

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION

Bureau of Educational and Cultural Affairs

Office of Policy and Evaluation

Evaluation Division:

English Language Evaluation Surveys

OMB Control Number: 1405-XXXX

SV2011-0031, SV2011-0032, SV2011-0033,

1. Justification

1. The Department of the State’s (DOS) Bureau of Educational and Cultural Affairs (ECA), Office of Policy and Evaluation, Evaluation Division (ECA/P/V) is requesting a new information collection to conduct a new ECA evaluation. ECA evaluations to date have provided significant evidence of the effect of ECA programs on the personal and professional achievement of participants, and have identified critical changes in the institutions where they have worked. This new evaluation will assess three different programs that fall under the realm of English Language within ECA in order to look at program outcomes.

1. An assessment of the E-Teacher Scholarship (E-Teacher) Program, will review the experiences of E-Teachers in the program, how they applied and shared what they learned at their home institutions, and how their experiences influenced their professional lives after concluding participation in the Program.

2. An assessment of the English Language Specialist Program, will review the roles and activities of Specialists in their assignments abroad, their effect at the institutional and country-level during these assignments, and how their experiences influenced their professional and educational choices after they concluded their participation in the program

3. An assessment of the Fulbright English Teaching Assistantship (ETA) program, will review the roles and activities of ETAs in their host institutions abroad, their educational and cultural effect in these schools and communities, and how their experiences influenced their professional and educational choices after they concluded their participation in the program

This information collection will include one survey sent to all participants who took part in one of these programs between 2004 and 2009. Because these are different programs with different foci, each program will be analyzed in distinct groups and not aggregated under one finding. The findings will also not be compared against each other, or used to generalize beyond those who respond to the surveys.

The data captured will help DOS and ECA Bureau successfully meet organizational performance and accountability goals established through the following mandates.

Further information can be found at the following links:

• Mutual Educational and Cultural Exchange Act of 1961, as amended (also known as the Fulbright-Hays Act) (22 U.S.C. 2451 et seq.)

http://www2.ed.gov/about/offices/list/ope/iegps/fulbrighthaysact.pdf

• Government Performance and Results Act of 1993 (GPRA)

http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html

• Government Performance and Results Modernization Act of 2010

http://www.whitehouse.gov/omb/performance/gprm-act

• OMB Increased Emphasis on Program Evaluations specified Memo

http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-01.pdf

As stated in the memo, “OMB will work with agencies to make information readily available online about all Federal evaluations focused on program impacts that are planned or already underway” as part of a three-pronged effort to strengthen government-wide program evaluation efforts. The guidance noted that public availability of program evaluation information will promote transparency, since agency program evaluations will be made public regardless of the results.

2. The primary purpose of this information collection is to provide ECA/P/V with the ability to assess the selected ECA programs in accordance with GPRA, as well as OMB Guidance, and Executive Orders. The data collected will inform the Program Offices in program management and future design issues or adjustments, program planning, results reporting, information dissemination and outreach initiatives.

As with all ECA evaluations, this study will examine the contributions of participants’ in select ECA Programs. It will provide State Department leadership, ECA senior management, and program officers with data they currently do not have, and analyses that can potentially be used to design new programs, improve extant programs, and to inform on-going and future activities.

This study will assess achievement of program goals only to the extent to which they are reflected in the major research questions below. The tables below list the major research questions developed for this evaluation per program, the outcome measures that may be assessed, and provide contextual information for understanding the affects of these three exchange programs. The data source that will be used to answer all of the major research questions will be via the set of on-line survey questionnaires.

Table 1

E-Teacher Scholarship Program
Major Research Question	Outcome Measure
What knowledge and skills have English teaching professionals gained through the E-Teacher Scholarship Program, in the following Teaching English as a Foreign Language (TEFL) courses or course areas: Critical Thinking in the EFL (English as a Foreign Language) curriculum Teaching English to Young Learners Building Teaching Skills through the Interactive Web English for Specific Purposes (ESP) Best Practices English as a Foreign Language Assessment, Various teaching methods courses	Answers to questions regarding participant learning and professional/academic development, including: acquisition of general TEFL pedagogies and best practices; more specific subject matter learning across each 6 different online course offering, and more general online/e-learning competencies gained through the online course experience.
How have English teaching professionals applied what they learned through the E-Teacher Scholarship Program, in their classes, schools, teaching institutions, or other organizational affiliations?	Answers to questions regarding changes or improvements in participant teaching practice (e.g. continued use and application of pedagogies discussed in course, use of materials, podcasts or web-based resources provided in course).
Has this led to changes or innovations in courses or curricula?	Answers to questions indicating resultant changes in participants’ courses or curricula (e.g. improved assessment practices, interactive/communicative activities).
Have they shared new knowledge and skills with their peers and colleagues?	Answers to questions about participant transmission of course-related knowledge to assess multiplier effect/wider reach of course on peers and colleagues in-country.
Through what mechanisms have they shared this knowledge? For example, through presentations, books, or articles in the discipline.	Answers to questions regarding how participants disseminate knowledge (e.g., workshops, presentations, books) and to what precise groups (their students, colleagues, other English language professionals).
In terms of their own professional development, how have English teaching professionals sought to expand their knowledge or improve practice since completing the E-Teacher Scholarship Program course?	Answers to questions about participant professional/academic development and further pursuit of knowledge following course experience (e.g. participation in additional online courses, attendance at major conferences, and professional exchange/study in the United States).
What is the “value-added” for participants who completed the E-Teacher Scholarship Program course? What gaps in their knowledge, skills or training did the course fill?	Answers to questions regarding professional development/training needs fulfilled by course.
How has access to an online English language education course provided E-Teacher Scholarship Program participants with opportunities they would not have had otherwise?	Answers to questions indicating personal and professional benefit of course to participants (e.g. interactions with U.S. experts, connection to international community of practice), contributions to professional development (e.g. course certificate, advanced graduate-level training), and influence of course participation on future professional development.

Table 2

English Language Specialist Program
Major Research Question	Outcome Measure
What has been the impact of English Language Specialist support in these areas? Curriculum projects Teacher training seminars Textbook development English for Specific Purposes (ESP), or Teaching program evaluation	Answers to questions regarding what specific types of support program participants extend to host-country English language professionals within primary areas of TEFL (Teaching English as a Foreign Language) best practices, specific pedagogies and learning/teaching materials (e.g. curriculum development, program evaluation, teacher training, conference plenary).
In what ways have English Language Specialists collaborated with universities, language institutes, or other host country institutions?	Answers to questions indicating how participants collaborated with and supported host country English language professionals and their institutions during and after their assignments (e.g. forming U.S.-host country institutional relationships, collaborating on training activities, supporting key English language teaching programs).
What roles have English Language Specialists played at conferences, workshops, or seminars, and to what effect?	Answers to questions identifying what activities participants have engaged in at formal in-country events/sessions (e.g. conferences, workshops, or seminars while on assignment in host-country setting).
How have English Language Specialists supported educational, organizational or institutional development in the host country? In schools, in communities, in teacher training institutions or universities?	Answers to questions indicating how program participants provide on-site assistance in the form of teacher training, program evaluation, ministerial consultation at varying levels (national, regional, local) or institutions (e.g. public, private, primary, secondary, tertiary) teaching locations.
How have English Language Specialists used teaching materials provided by the ECA Office of English Language Programs to support their teaching-related activities? Prior to the start of the assignment During the assignment After the assignment	Answers to questions indicating whether and how participants use or distribute specific ECA teaching materials (e.g. using materials in trainings or presentations, distributing or recommending materials to English language professionals).
How has the English Language Specialist Program facilitated reciprocal relationships and encouraged shared learning and knowledge, among English Language Specialists, host country individuals, or collaborating institutions?	Answers to questions regarding changes in participant understanding and knowledge of host countries, including: whether formal exchanges were developed, joint research publications were generated, and people-to-people networking or visits conducted.
How has participation in the English Language Specialist Program provided English Language Specialists with opportunities they would not have had otherwise?	Answers to questions regarding participant’s own added professional development obtained via the program including: enhanced professional reputation, increased understanding of TEFL-related issues world-wide, etc.
How have English Language Specialists benefited as individuals (professionally and personally) from their participation in the English Language Specialist Program?	Answers to questions regarding participant’s value-added in terms of significant new insights into different countries, societies, and cultures, understanding of differential challenges and approaches to TEFL, etc.
How does the English Language Specialist Program foster mutual understanding and strengthen relations between the United States and its citizens, and other countries and their citizens overseas?	Answers to questions regarding exchanged learning opportunities among program participants and other international TEFL professionals, creation of a community of practice, sharing of professional and pedagogical challenges, etc.

Table 3

Fulbright English Teaching Assistantship (ETA) Program
Major Research Question	Outcome Measure
What language teaching experience did the teaching assistant gain as a result of participating in the program?	Answers to questions regarding participant learning and professional/academic development (e.g. What kinds of teaching activities participants engaged in and the experience, such as classroom management and lesson planning, gained through these activities.)
How does the Fulbright English Teaching Assistantship (ETA) Program foster mutual understanding and strengthen relations between the United States and its citizens, and other countries and their citizens overseas?	Answers to questions about how participant introduced U.S. values, culture, and lifestyle to host countries through their teaching and other activities. Also, answers to questions about participant exposure to foreign, culture, languages, values, and lifestyle and how this exposure changed their choices and perspective.
How did host country teachers and language instructors gain from having an English language teaching assistant in their classrooms?	Answers to questions indicating how participants collaborated with and supported host country teachers in the classroom (e.g. Whether and how they assisted with classroom teaching and transferred new skills to host country teachers.)
To what extent, and with what outcome, did participants communicate information about U.S. culture, history, and society?	Answers to questions regarding extent to which participants’ students gained better understanding of U.S. history, society, culture, and contemporary issues (e.g. American traditions, holidays, popular culture, literature, sports, and religious freedom) as a result of participant activities.
What role/roles did the teaching assistant play in the class room?	Answers to questions identifying the primary activities participants engaged in the classroom (e.g. teaching English language skills, practicing conversation, communicating U.S. history and culture)
Did host institution teachers take the opportunity to act as mentors for their teaching assistants?	Answers to questions asking how host institution teachers supported participants in the classroom (e.g. helping develop classroom management skills and new teaching techniques, and choosing appropriate teaching materials)
To what extent does the Fulbright ETA Program facilitate reciprocal relationships and encourage shared learning and knowledge?	Answers to questions addressing extent to which participants have stayed in contact with their students, fellow teachers, and others.
Did the teaching assistants face challenges? If so, how did they deal with these challenges?	Answers to questions identifying challenges faced by participants, such as adjusting to a different culture, heavy workloads, lack of teaching support from host institution, or handling daily logistics.
Did participation in the program inform plans for graduate study, career plans, or other professional goals?	Answers to questions regarding participants’ professional and academic choices following their assignment, such as how the program has helped them obtain new employment opportunities or expanded their skill sets, and whether or not they have continued in the field of English language teaching.
How has participating in the Fulbright ETA program created opportunities that would not have existed otherwise?	Questions about how experience in the program has changed participants, including giving them experience in the classroom, changed their understanding and perspective on other countries and cultures, and provided them new skills and employment opportunities.

Analysis of all collected data will include descriptive statistics and frequencies, providing percents of scores and counts per each response category for the survey questions relating to each research question. Cross-tabular analysis of survey responses will be conducted to assess variances in effect by different participant or program characteristics. Examples may include comparing program-level findings by type of professional engagement (classroom teachers vs. administrators) or by school level/type (primary, secondary, post-secondary), We also anticipate analyzing differences in particular participant-level effects (e.g. satisfaction, continued engagement with host-country peers) across different cohorts to determine how these effects differ relative to the length of time since program completion. We will also not report any finding when n is less than or equal to 5 in order to protect respondent confidentiality and to ensure we are not reporting invalid results

3. The information collection surveys will be entirely web-based to ease any burden on the participant. The survey will be distributed using the survey application Vovici.

4. Currently, no duplicative information exists, and there is no other reliable method for ECA to collect the information needed to fulfill the requirements of the Department’s annual strategic planning and reporting process and the annual Congressional budget process as part of the GPRA, PART and PMA mandates.

5. Information collected under this collection will have no impact on small businesses and other small entities.

6. If the information is not collected, ECA will be unable to complete this study, or gather data requested by ECA senior leadership in order to assess and report on these types of English language exchange programs which are part of the new strategic plan of ECA. Moreover, the Department will be unable to comply fully with its congressional and DoS executive mandates, including the GPRA Modernization Act of 2010 which requires the agency to evaluate and report the results of its exchange programs. There are no technical or legal obstacles to reducing burden.

7. There are no special circumstances.

8. ECA/P/V has solicited public comments on this collection via a 60-day Notice published in the Federal Register on February 7, 2012 (77 FR 6168) seeking public comments. One comment was received. Upon reviewing the comment, ECA/P/V determined that the comment was unrelated to the information collection and instead addressed broader Department wide policy and budget regarding the implementation of the programs, and the Fulbright-Hays Act supporting ECA–related exchange programs. ECA/P/V has consulted with an external contractor, EurekaFacts about the surveys design, methodology, analysis, and data collection approach.

9. No gifts or payments will be made to the respondents.

10. The agency keeps information private to the extent permitted by law. This will be reflected on all surveys and survey messaging.

11. No questions of a sensitive nature are asked in the survey.

12. It is estimated that the total annual hour burden will be 1,077 hours for the 3,400 respondents that make up the census population (As explained in Section B1, it is estimated that the response rate to the surveys will be 40%.) The annual hours burden was calculated with the expectation that 40% will complete each survey at 40 minutes (907 hours), and 60% of the population is estimated to not respond to the invitations (at an estimated average of 5 minutes) for 170 hours of burden. Because this survey will only be conducted once, the three year total is the same as the annual total.

Table 4

Respondent Burden per Survey

	ITEM	ANNUAL TOTAL	3 YEAR TOTAL
E-Teacher Scholarship Survey	Estimated Number of Respondents	800	800
	Average Hours Per Response	40 Minutes	40 Minutes
	Estimated Number of Responses	320	320
	Estimated Number of Non-Responses	480	480
	Estimated Hours for Responses	213	213
	Estimated Hours for Non-Responses	40	40
	Estimated Total Hours- E-Teachers	253	253
English Language Specialist Program Survey	Estimated Number of Respondents	250	250
	Average Hours Per Response	40 Minutes	40 Minutes
	Estimated Number of Responses	100	100
	Estimated Number of Non-Responses	150	150
	Estimated Hours for Responses	67	67
	Estimated Hours for Non-Responses	12.5	12.5
	Estimated Total Hours- ELS	79.5	79.5
ETA Survey	Estimated Number of Respondents	2,350	2,350
	Average Hours Per Response	40 Minutes	40 Minutes
	Estimated Number of Responses	940	940
	Estimated Number of Non-Responses	1410	1410
	Estimated Hours for Responses	627	627
	Estimated Hours for Non-Responses	117.5	117.5
	Estimated Total Hours- ETA	744.5	744.5
Total Hours		1,077	1,077
Total Responses		1,360	1,360
Total Respondents		3,400	3,400

Each survey was pre-tested prior to this submission using five (5) former participants. Burden hours took this into account, as well as the total number of questions and the number of open-ended questions, as well as experience on previously conducted evaluations.

13. There are no costs incurred by respondents.

14. The data collection budget for this evaluation survey is approximately $126,000. This includes contractor labor for drafting the survey as well as the actual survey administration/data collection and software/server expenditures. The analysis and reporting budget for the data collected through this collection is approximately $78,000, and will include contractor labor for analysis, report writing and materials, and briefings

It is therefore estimated that the annualized cost to the Federal government for this collection will be $204,000.

15. This is a new collection.

16. Survey data collection is estimated to begin immediately after OMB clearance approval is received. It is estimated the data collection period will take at least 6 weeks. Following the data collection period, the external contracting firm (EurekaFacts) will conduct basic descriptive analysis (such as frequencies) and cross-tabular analysis as needed as explain per section A2. A report will be developed, with several iterations submitted to ECA for review and approval. Once approved by the Assistant Secretary of the Bureau, ECA/P/V will release the final evaluation report to the public.

The evaluation report will be posted on the DOS ECA Evaluation Division site at http://exchanges.state.gov/programevaluations/completed.html for wide public release. Additionally, an appropriate distribution list, which will include key stakeholders and other organizations and individuals that may be interested in the evaluation results, will also be developed. They will receive notification of the release of this report via email. The contracted evaluators are also required to present results of the evaluation to key stakeholder groups as requested by ECA for a period of time following the evaluation’s completion. Results for this evaluation are estimated to conclude about 9 months after the data collection period has ended.

17. ECA/P/V will display the OMB expiration date.

18. There are no exceptions requested for this collection.

2. Collections of Information Employing Statistical Methods

1. There is no sampling for this information collection, as the potential respondent universe for this information collection will be all 3,400 program participants from the E-Teacher, ETA, or English Language Specialist Programs. E-Teacher participants are foreign citizens living in their home countries, while ETA and English Specialist are U.S. citizens. Participants surveyed will cover the years between 2004 and 2009. The participants from these program years have never been evaluated in regards to these research questions before. The anticipated response rate for this entire collection is 40%. This number is based on experience with previous DoS studies that have been completed, the response rates in a currently underway study, and on contacts with the program office and the grantee organization.

2. This information collection will consist of three electronic surveys. Only the applicable survey will be administered to participants from that program. Because of the duration that has passed since participation in the program, the importance of obtaining sufficient responses for each program year, and the low total N in the programs we’re surveying, sampling would likely yield an insufficient number of responses. For the one program in which sampling may be appropriate (the ETA program, with N>2000) probability sampling would be difficult due to potential concerns with contact information for participants from earlier cohorts. As a result, the statistical methodology used will be via census. This information collection will only be conducted one time as part of the English Language Evaluation.

3. All ECA/P/V data collection methods are tailored to fit the prevailing political, cultural, safety, security, and accessibility conditions in each country in which participants are located. Successfully contacting and achieving the highest possible response rates are the goals of survey administration. Our methods will include:

• Customized Intro Email: A customized intro email will be sent at the start of survey administration to encourage respondent cooperation. This email will inform them about the evaluation and will also provide ways for respondents to contact the evaluation’s contractor with any concerns or questions about the evaluation.

• Participant Contact Information Verification: Extensive contact lists for each program were requested from the respective administering grantee organizations and State Department program offices to establish baseline participation in each program over the 2004-2009 period and to obtain an initial set of contact data. In addition, ECA/P/V queried the State Department’s Alumni databases to obtain any additional or updated contact information in order to ensure that the contact lists are as accurate as possible.

• Informing the Grantee Organizations Prior: Many program participants continue to be in communication with the grantee organization that administered their exchange program long after the program has ended. Informing the grantee organizations in advance of the start of the evaluation’s data collection period will allow the grantees to vouch for the survey requests that get sent out by the contractor. Doing this will only serve this purpose in the event any of the participants contact the grantee regarding any doubt as to the legitimacy of the initial intro email that will be sent by EurekaFacts. No other information about the participants themselves will be provided to the grantee.

• Survey Reminders: Besides the initial intro email, three follow-up reminders will be sent to non-respondents to encourage them to respond over the course of the administration period, including a final reminder as the survey comes to a close that will indicate the urgency. Response rates and survey user feedback will be monitored and recorded upon each biweekly reminder to ensure a satisfactory response. ECA/P/V will also be ready to make a judgment call based on response rate status throughout the administration period to both extend the administration period as deemed fit, as well as send an additional reminder.

• Pre-testing Survey: Pre-testing the survey was extremely useful for clarifying instructions and questions, refining the response categories, as well as ensuring clarity, brevity, relevance, user-friendliness, understandability, and sensitivity to a respondent’s culture and the political climate in which they live. This in turn allowed the survey’s questions to be designed in a way in which to minimize the burden to respondents and encourage them to complete their survey.

Using such methods has in our previous experiences stimulated response rates.

This data collected is only representative of the evaluation’s respondents and all analysis of results and future reports will be clearly linked to only the universe that was surveyed. We will monitor the potential for non-response bias, including tracking response rates by cohort over the collection period and reviewing both respondent and non-respondent demographics. These factors will be taken into account in our analysis and reporting of results, especially when disaggregating the data according to key demographics for which the number of respondents may be less than ideal.

4. To enhance each questionnaires design, a small number of formative interviews were conducted. For each survey, five (5) former program participants were interviewed prior to the survey development phase. Each program’s questionnaire included a distinct set of questions, thus complying with the PRA information collection requirement that identical questions were not asked of 10 or more respondents. These interviews increased each questionnaire designers’ level of understanding in regard to program participants’ experiences, particularly in terms of identifying the full range of activities, interactions, roles, and outcomes associated with program participation. In addition to formative interviews prior to questionnaire design, a small number of cognitive/pre-test interviews were conducted upon completion of the questionnaire design phase. As part of these interviews a small number of past program participants, completed a test version of the on-line survey and were later de-briefed through telephonic interviews or via e-mail to identify any needed modifications to the instrument prior to OMB submission. The debriefing interviews focused on determining whether question wording was clear, conveyed its intended meaning, contained realistic and mutually exclusive response options, and presented scaling of magnitude, agreement/disagreement, etc. that is relevant and understandable to the respondents.

5. The ECA/P/V individual managing this evaluation’s external contractor (EurekaFacts) who will be collecting the data and analyzing the information is Julien Kreuze, 202-632-6317.

10