Download (844 KB)

(1)

WEI SURVEY OF PRIMARY SCHOOLS

TECHNICAL REPORT

(2)

UNESCO

The constitution of the United Nations Educational, Scientific and Cultural Organization (UNESCO) was adopted by 20 countries at the London Conference in November 1945 and entered into effect on 4 November 1946. The Organization currently has 193 Member States and 7 Associate Members.

The main objective of UNESCO is to contribute to peace and security in the world by promoting collaboration among nations through education, science, culture and communication in order to foster universal respect for justice, the rule of law, and the human rights and fundamental freedoms that are affirmed for the peoples of the world, without distinction of race, sex, language or religion, by the Charter of the United Nations.

To fulfil its mandate, UNESCO performs five principal functions: 1) prospective studies on education, science, culture and communication for tomorrow's world; 2) the advancement, transfer and sharing of knowledge through research, training and teaching activities; 3) standard-setting actions for the preparation and adoption of internal instruments and statutory recommendations; 4) expertise through technical co-operation to Member States for their development policies and projects;

and 5) the exchange of specialized information.

UNESCO is headquartered in Paris, France.

UNESCO Institute for Statistics

The UNESCO Institute for Statistics (UIS) is the statistical office of UNESCO and is the UN depository for global statistics in the fields of education, science and technology, culture and communication.

The UIS was established in 1999. It was created to improve UNESCO's statistical programme and to develop and deliver the timely, accurate and policy-relevant statistics needed in today’s increasingly complex and rapidly changing social, political and economic environments.

The UIS is based in Montreal, Canada.

Published in 2009 by:

UNESCO Institute for Statistics P.O. Box 6128, Succursale Centre-Ville Montreal, Quebec H3C 3J7

Canada

Tel: (1 514) 343-6880 Fax: (1 514) 343-5740

Email: [email protected] http://www.uis.unesco.org

ISBN 978-92-9189-082-8 Ref: UIS/TD/09-08

(3)

Acknowledgements

This technical manual was co-ordinated and edited by Ursula Itzlinger (UIS). Special appreciation is also given to the various authors, revisers and contributors, including:

Aletta Grisay (University of Liège), Sonia Gontero (UIS), Nicola Melki (UIS), Albert Motivans (UIS), John Pacifico (UIS), T. Neville Postlethwaite (University of Hamburg), Owen Power (Statistics Canada), Mamadou Thiam (UIS), Hélène Tran (UIS) and Yanhong Zhang (UIS).

(4)

(5)

Chapter 1. Overview

1.1 Survey background

The World Education Indicators (WEI) programme was founded in 1997 as a joint endeavour of the UNESCO Institute for Statistics (UIS) and the Organisation for Economic Co-operation and Development (OECD). It focuses on consolidating basic education statistics, in addition to launching special projects to collect additional data on current and emerging education issues.

The WEI Survey of Primary Schools (WEI-SPS) was established in 2002 to address education quality and equitable distribution among students by producing internationally comparable data on the functioning of schools, teaching, learning conditions, and resources available to support change both in education systems and the communities they serve. In 2005 and 2006, 11 of the 19 countries in the World Education Indicators¹ programme participated in this survey, which collected detailed information on the context, conditions and conduct of primary schools. For some participating countries, the study also presented an opportunity for capacity building to support large-scale national assessments.

The data from this survey has enabled researchers to answer many of the questions surrounding key issues in education (UNESCO-UIS, 2008).

1.2 Survey objectives

The main objective of the WEI-SPS study was to obtain cross-national data on how schools function, including the amount of resources available to them. The participating countries used the data collected to explore questions on school inputs, policies and processes. The countries were also interested in learning the extent to which resources and good practices were equitably distributed among schools.

Specifically, participating countries wanted to address the following questions:

• How do the contexts of primary schools compare across countries? What material and human resources were available, and how did they compare across countries? How equitable is the distribution of these resources within countries?

• What are the basic characteristics of the pupil population served by primary schools in each country? What are the transition patterns at the end of primary school and between the primary and lower secondary levels?

• To what extent do countries vary in the official number of school days per year and the number of lost school days per year? To what extent and at what level is educational leadership provided in schools, both across and within participating countries?

• How much do countries vary in the way that reading and mathematics are taught and in the emphasis placed on different aspects of these two subjects?

(8)

• To what extent is the school climate conducive to acceptable levels of instruction, learning, achievement orientation and discipline across countries? How equitable is the distribution of these conditions within countries?

• How do countries compare in terms of teaching strategies and styles?

1.3 WEI-SPS framework

The focus of the WEI-SPS study was on equality, efficiency and quality of learning provided in primary schools. In order to keep the survey concise, emphasis was placed on indicators that had been described in literature as being significantly associated with school and instructional effectiveness and which could be operationalised on the school and classroom level. Within this selected group of indicators, factors related to malleable conditions at the school and classroom levels (i.e. conditions that can be changed by the actions of the school or an outside agency) were highlighted. These factors can be grouped into three distinct categories:

i. Context: The environment in which individual schools operate. For example, the broader legal and regulatory context in a country determines the role and impact of private schools in the education system.

ii. Input: The material and human resources available to schools. These resources range from budget allocations to the experiences and qualifications of teachers.

iii. Process: The transformation of input into output. Processes take place at several levels, including country or system, school, classroom and the individual learner (Scheerens, 2002a).

1.4 Survey design and administration

Although the countries participating in WEI-SPS were equally interested in a survey of schools at the upper secondary level, it was decided to focus on the primary level in order to yield more comparable data and it was easier to implement. The survey design and framework were created in a collaborative effort by participating countries, the OECD and the UIS.

In 2005 and 2006, the main survey of WEI-SPS was conducted in 11 countries (see Table 1). International sampling experts oversaw and approved – and in some cases implemented – the national sampling plans. Countries could sample more schools to yield more accurate results but had to select at least 400 schools as a baseline. In many countries, a larger sample was used to obtain more information on areas of national interest, most notably by region, province or school sector (i.e. public or private schools).

(9)

Table 1. WEI-SPS participating countries

Country Schools eligible Schools sampled

Argentina 16,900 1,189

Brazil 127,441 646

Chile 6,103 668

India² 6,103 668

Malaysia 6,796 742

Paraguay 5,537 812

Peru 5,537 812

Philippines 5,537 812

Sri Lanka 7,255 612

Tunisia 3,758 470

Uruguay 1,453 437

In several countries, more than one language of instruction is used in primary grades, either throughout a country or even within a school. The questionnaires had to be delivered in a language in which school heads and teachers would be proficient (e.g. the language used in the teacher’s training). This language did not necessarily have to match the language in which the students are taught. An international English version of the questionnaire was translated into a total of eight languages, Spanish being the most common among participating countries. To ensure international comparability, translation verification of all instruments was performed in each language.

International standardisation of data collection and data entry is a further means to ensure comparable data. The geographic, cultural and economic diversity found in the group of WEI-SPS countries presented a challenge in creating international standards for operational issues. As a result, operational standards were detailed in a series of manuals, and software tools were implemented for standardised and quality-controlled data entry.

The school head, all Grade 4 teachers of mathematics/arithmetic and reading, and national curriculum experts were defined as respondents. In most countries, teachers at the primary level tend to be home-class teachers who teach the whole class in both subjects. The survey was administered near the end of the school year in all countries, since questions sought information on student backgrounds and perceived academic performances.

Four questionnaires were developed to be completed in less than one hour. Surveyors distributed and collected materials, in addition to ensuring that data were correct and complete. One or more national curriculum expert per country completed the curriculum questionnaire.

With support from the UIS, data were entered and cleaned in the participating countries.

Subsequently, a database was finalised containing only internationally comparable data.

Indices and composite variables were created based on research literature and data analysis, conducted at the UIS and featured in an international report (UNESCO-UIS, 2008).

(10)

1.5 Survey management

The survey was designed and implemented jointly by the OECD and the UIS, aided by a network of consultants and international experts. Each participating country assigned a National Project Manager (NPM), who was responsible for the implementation of the survey at the country level and liaised with the international platform. A Steering Committee, consisting of WEI country representatives, international experts and officials from the OECD and the UIS (see Appendix I), was established early in the process to guide survey development, priority findings and methodological discussions, in addition to approving the survey design.

The OECD and the UIS managed the implementation of the survey at the international level. The OECD team was in charge of framework and instrument development, as well as contracting international experts to aid in this effort. The UIS team provided operational guidelines and led the sampling, which also entailed contracting international school sampling experts. The UIS compiled the international database, led the data analysis and reporting, and conducted workshops for data analysis. All international survey costs and quality control costs were born by the OECD and the UIS, with financial support from the World Bank. The NPMs were responsible for national implementation, including finding research questions of national interest, defining the national sample together with the international sampling experts, budgeting, and national reporting.

1.6 Organization of the report

In order to judge the quality, accuracy and reliability of a survey, readers require access to background information. Data users need details on the meaning of data, what weights are available and limitations to the data. The goal of this report is to describe the technical aspects of the survey. Analytical findings are not included in this report. They can be found in the UIS report, A View Inside Primary Schools: A World Education Indicators (WEI) Cross-National Study (UNESCO-UIS, 2008).

Chapter 2 describes the framework of the survey and the process of instrument development. Chapter 3 explains the necessity of quality control in translating international questionnaires. It describes the processes to which all translated texts were subjected, in order to ensure that standardised instruments were available in the appropriate national languages.

Chapter 4 gives an introduction to the sampling issues of WEI-SPS. It lists the definitions for the target population, stratification, non-response and response rate standards.

Chapter 5 describes in detail the samples desired and achieved in the participating countries, including exclusion rates. It shows the calculation of weights: school weights, teacher weights, student-based school and teacher weights, adjustment for non-response, and benchmark-adjustment for the weights.

Chapter 6 deals with the administration of the survey, as well as the guidelines and

(11)

Chapter 2. Questionnaire development

Proper schooling is a crucial element in any effort to improve the quality of learning. What makes some schools or school systems more successful than others in equipping their students with an equitable education of high quality? How do primary schools in WEI countries compare, both across and within countries? These are some of the questions that sparked the creation of the WEI-SPS.

The development of the WEI-SPS questionnaires was headed by the OECD, with support from the UIS and assistance from experts in the fields of education and education research. The experiences of diverse organizations and projects were taken into account (see Section 2.2). Numerous consultations were needed during the development phase and for selection of questions to address. In June 2003, participating countries rated potential indicators in terms of priority and a framework for the survey was established.

After further discussions at WEI conferences and several meetings of the SPS Steering Committee, the framework and the questionnaires were finalized.

The final questionnaires in English were distributed to countries for translation and to initiate the pilot study, which took place in the second half of 2004. After a final priority rating of certain questions in November 2004 and an analysis review in December 2004, the main survey questionnaires were developed (OECD-UIS 2005b).

2.1 Analytical framework and indicators

At the core of WEI-SPS lies the analytical framework. Based on previous research on school effectiveness (which is summarised in Figure 1), it defines the context of the survey and the research questions used. The framework also serves as the scaffold for the questionnaire development.

The framework outlines the conceptual basis for the selection of indicators on the functioning of schools, in particular concerning school effectiveness and equity (Scheerens, 2003). The choice of indicators took the following points into consideration:

• The purpose of the proposed SPS was to provide internationally comparable indicators on the quality and equity of schooling among and within the WEI countries;

• These indicators should be considered relevant to audiences at various levels of national systems: government officials, regional or provincial officers and school directors. Feeding back information in a format that was considered useful to each of these audiences was a major objective of the survey; and

• The school survey encouraged participating countries to add national questions, enhancing the relevance for country-specific needs. The possibility of using the survey as a more permanent instrument of educational monitoring and information provision was taken into consideration.

(12)

Figure 1. Summary of the research findings on school effectiveness

Source: Scheerens, 2002a.

(13)

Indicators at school level – School Questionnaire

The School Questionnaire was completed by the school head. Most questions refer to the entire primary school, including all primary grades. A few questions (e.g. instructional time in Grade 4 classes) were intended for a specific grade only. The indicators at school level for the School Questionnaire are:

• School material resources

• Availability and condition of school resources – school facilities

⇒ Availability of school resources – school materials

⇒ Adequacy of school resources – principal's perceptions of shortages in school material resources

• School human resources

⇒ School human resources – staff

⇒ School size and class size

⇒ Staff qualification

⇒ Staff stability

⇒ Permanent and temporary teachers, support staff

• Educational leadership

⇒ Principal’s workload

⇒ Principal’s activities

⇒ Educational leadership (principal’s management rating scale)

• Curriculum alignment

⇒ Curriculum alignment

⇒ Instructional time

⇒ Instructional time – days lost

• Staff consensus and cooperation

⇒ Frequency of staff meetings

⇒ School staff co-operation – practices

⇒ School staff co-operation – official policy

⇒ Principal's relations with teachers

⇒ Staff Consensus

• Parental involvement

⇒ Parents and community contributions

⇒ School - parent relations

• School autonomy

⇒ School autonomy - participation in school decision making

• Professional development of school head and staff

⇒ Staff professional development

⇒ Principal's professional development

⇒ Types of professional development activities

⇒ Proportion of staff involved in several kinds of professional development activities

• School evaluation

⇒ School assessment practices – the different uses intended for student assessments

⇒ School evaluation and teacher appraisal

(14)

• School climate

⇒ Principal's perception of staff morale

⇒ Principal's perception of students' motivation

⇒ Behavioural problems of students

⇒ School disciplinary climate concerning teacher behaviour

• Ability grouping and school remedial activities

⇒ Ability grouping between classes/streaming

⇒ Ability grouping – grouping within classes

⇒ School remedial – extension activities

• School ambition

⇒ Extra activities

⇒ Principal's locus of control

• Indicators on school background characteristics

⇒ School type – governance

⇒ School type – funding

⇒ Community in which the school is located

⇒ School admission policies

⇒ Distance of the school to public resources and facilities

⇒ Background characteristics of the school's intake – general conditions relative to "being disadvantaged"

– socio-economic status

– linguistic background and discrepancy between the language spoken at home and at school

• School structure

⇒ Grades taught

⇒ Shifts

⇒ Multi-grade classes

⇒ School sites

• School attrition

• Programme completion

Indicators at classroom level – Teacher Questionnaire

The Teacher Questionnaire was completed by mathematics/arithmetic or reading teachers of Grade 4 classes in a school, with respect to their instructional environment in these Grade 4 classes. The indicators at classroom level for the Teacher Questionnaire are:

• Instructional resources

⇒ Classroom furniture - tables and chairs

⇒ Classroom equipment

⇒ Textbooks

• Instruction time in basic subjects

⇒ Official instruction time - language

⇒

(15)

• Student assessment at classroom level

⇒ Assessment methods

⇒ Relative importance of different assessment methods

⇒ Use of student assessment

• Active learning

⇒ Active teaching in reading

⇒ Active teaching in mathematics

⇒ Reproductive and active learning activities

• Differentiation

⇒ Internal differentiation in instructional approach and grouping

• Teacher professional satisfaction

• Structured teaching/scaffolding

• School goals and achievement expectations

• Indicators on teacher background characteristics and attitudes

⇒ Descriptive background characteristics

– Gender

– Age

– Experience as a teacher – Level of education

– In-service training

⇒ Perception of working conditions – Self-perception in terms of status

⇒ Class size, number of shifts.

Indicators at classroom level – Annex to the Teacher Questionnaire (OTL)

The Opportunity to Learn (OTL) Questionnaire was completed by the same teachers who took the Teacher Questionnaire. Subject teachers answered only for the subject they were teaching in Grade 4. In all WEI-SPS countries except in Malaysia, it is unusual to have subject teachers at the primary level; most teachers are home class teachers who teach all or most subjects. The indicators at classroom level for the Annex to the Teacher Questionnaire are:

• Opportunity to learn in reading and mathematics/arithmetic

⇒ the level of difficulty of the reading materials that classroom teachers consider appropriate for use in an “average” Grade 4 class

⇒ the types of reading materials used most often

⇒ the nature of the reading questions and reading activities that are typically used for Grade 4 students

⇒ the level of difficulty of the Mathematics activities that classroom teachers consider appropriate for use in an “average” Grade 4 class

⇒ the nature of the Mathematics tasks that are typically used for Grade 4 students.

(16)

Indicators at system level – National Project Manager (NPM) Questionnaire (or Curriculum Questionnaire)

The NPM Questionnaire was completed by national curriculum experts. They were asked questions on the national education system. Indicators at the system level are:

• Opportunity to learn in reading and mathematics/arithmetic (same as in the Annex to the Teacher Questionnaire)

• Education system indicators

• The teaching force

• Current reforms affecting Grade 4 education, if applicable.

It should be noted that not all indicators could be included in the main WEI report (UNESCO-UIS, 2008). However, the international WEI-SPS database includes the data necessary for analysis of these indicators.

2.2 Piloting the questionnaires

Following the assessment of the relevance and policy priority ranking of the indicators in the WEI participanting countries, pilot questionnaires were constructed using established and tested questions as much as possible. The pilot surveys used questions drawn from international and national surveys, notably from those of the International Association for the Evaluation of Educational Achievement (IEA), the Organisation for Economic Co- operation and Development (OECD), the School Achievement Indicators Program (Canada), the Schools and Staffing Survey (United States), the Southern and Eastern Consortium for Monitoring of Educational Quality (SACMEQ), Zelfevaluatie basisonderwijs – ZEBO (Self-Evaluation in Primary Education, the Netherlands), the Victorian Department of Education (Australia) and the Assessment Research Centre (Australia). If no adequate question could be found, a questionnaire design team drafted the questions specifically for the survey.

In November 2003, four questionnaires were sent to countries for review:

• School Questionnaire to be completed by the school head;

• Teacher Questionnaire to be completed by all teachers in Grade 4 of the surveyed school who teach reading and/or mathematics/arithmetic at this school;

• Annex to the Teacher Questionnaire (Opportunity To Learn-Questionnaire, OTL) to be completed by the same teacher as the Teacher Questionnaire; and

• National Project Manager Questionnaire (Curriculum Questionnaire) to be completed by curriculum experts in the participating countries.

In Argentina, Chile, Malaysia, Peru, the Philippines and Tunisia, the questionnaires were

(17)

Table 2. Pilot samples and case numbers

Country No. of school

questionnaires No. of teacher

questionnaires OTL version A OTL version B

Brazil 40 85 40 46

Chile 19 26 13 11

India 120 137 - 136

Indonesia 193 174 96 97

Malaysia 76 247 139 98

Paraguay 43 71 43 28

Peru 29 30 - 30

Philippines 20 62 26 36

Sri Lanka 35 60 58 58

Tunisia 11 39 - -

Uruguay 23 39 21 21

Total 609 970 436 561

Due to scheduling problems, Argentina did not participate in the pilot but tested the operations using the main survey questionnaires in spring 2005 in six schools. Due to financial problems, Indonesia dropped out of the survey after the pilot. Tunisia had decided earlier not to administer the OTL questionnaire in order to keep the survey time as short as possible. India and Peru administered version B of the OTL questionnaire only in the pilot.

In the main survey, only version B of the OTL questionnaire was used internationally, as this version showed more variation across countries. Items where the modal grade was not Grade 4 for all countries were replaced with items from version A of the OTL questionnaire (OECD/UIS 2005c).

The NPM (Curriculum) Questionnaire was piloted in Brazil, India, Paraguay and Uruguay.

In Brazil and India, it was administered to the curriculum experts at the provincial or regional levels since the curricula in both countries varied by region/state or even municipality. In the main survey, the NPM Questionnaire was administered in all countries.

In most countries, it was administered to a group of curriculum experts, who completed one questionnaire together. In countries with federal structures, this proved to be difficult, as the curriculum experts of the respective regions reported that their curriculum differed substantially. In the case of Brazil and India, the resulting “national intended curriculum” is artificial even though the data of the pilot showed overlaps for a number of curriculum issues. In India, the Curriculum Questionnaire was administered to curriculum experts of the participating states only (see Chapters 4 and 5 for details on participating states).

(18)

2.3 Finalizing the main survey questionnaires

Upon receiving all data, an international pilot database was created and analysis commenced. This process included analysing the feasibility and validity of questionnaire items across countries and verifying that the material was cross-culturally appropriate. For example, when National Programme Managers validated the questionnaires prior to piloting, a few questions had to be omitted as they were deemed inappropriate in some participating countries. This affected mostly OTL questionnaire items.

Results from the pilot indicated that the questionnaires were too long, especially the School Questionnaire. While reviewing the data, problems with some of the questions were also detected. Due to time constraints, restructuring and further piloting was not an option and, as such, the questions that would have required major changes or re-piloting were dropped from the questionnaires. Wherever possible, the priority ranking of items were taken into account when dropping questions. In the School Questionnaire, some questions were made optional. They were moved to the end of the questionnaire and could be administered by those countries that had deemed them as high priority (OECD/UIS 2005a, 2005b and 2005c).

In addition, minor changes were implemented to clarify some questions. In some cases, answer categories were added or collapsed to improve future analysis if the pilot results indicated insufficient discrimination between answers from two categories, or if there were too few respondents to choose a particular category.

Questions regarding national school systems and teaching forces were deleted from the NPM Questionnaire, as there were other sources where the data could be found, notably the UIS and IBE databases.

Results from processing the pilot data also triggered further amendments to the main survey questionnaires. For example, questions that generated a high number of missing data or outliers, or failed some of the logical tests, were assigned for further review by the questionnaire design group. In addition, all open-ended questions were deleted or replaced with categorical questions.

When data were of inconsistent or implausible values, they were not cleaned automatically. In the pilot, these cases were only listed and country NPMs were presented with the list of inconsistencies. These lists helped identify problematic questions, which were then reviewed and eliminated or rephrased.

(19)

Chapter 3. Translation procedures

Survey instruments were developed in English and later translated by WEI-SPS participating countries into their main language(s) of instruction. The instruments were translated into a total of eight languages and adapted in English for one country. Out of the 11 participating countries, one administered the survey in three languages while another administered it in two. The most common language in which the survey was administered was Spanish (five countries), and the second most common language was Tamil (two countries). Details on the languages used can be found in Table 3. Countries adapted and translated the instruments following the guidelines compiled in Instructions for Translation/Adaptation of the WEI-SPS Main Survey Questionnaires (OECD/UIS/WEI/SPS(2005)9).

Table 3. Language used for the SPS survey

Country Languages

Argentina Spanish Brazil Portuguese Chile Spanish India Assamese, Hindi, Tamil

Malaysia Bahasa Malaysia

Paraguay Spanish Peru Spanish Philippines English Sri Lanka Sinhala, Tamil

Tunisia Arabic Uruguay Spanish

The translated survey instruments were thoroughly reviewed and verified before being used in the field for administration. The UIS oversaw and coordinated the process between NPMs and verifiers to ensure international comparability of the instruments. The Institute also used additional verifiers from a linguistic quality control firm and its own staff to review each translation.

The verification process was iterative until final approval by the UIS. First, the NPMs submitted their documents to the UIS. The UIS reviewed the documents and commented on them in the National Adaptation and Translation (NAT) form (see Appendix II). The verifiers would then review the instruments, document any deviations from the international versions in the NAT forms and answer queries from the UIS that were inserted in the NAT forms. The UIS again would review the documents before sending the questionnaires, along with the NAT forms, back to the NPM for corrections or explanations. The verification process continued until the UIS provided the approval to the NPM to print and administer the documents.

3.1 Translations

The OECD designed the translation verification procedure and the translation and adaptation manual with the associated forms, based on the procedures and materials used for PISA questionnaires. The UIS was responsible for reviewing and approving national adaptations and verifying the international comparability of the instruments. Countries

(20)

The objectives of the translation and adaptation guidelines were to:

• Describe procedures to translate and adapt WEI-SPS questionnaires into national language(s) and national cultural context;

• Describe acceptable modifications that could be made to the instruments as part of the translation/adaptation process; and

• Document procedures for submitting the instruments for adaptations approval and for international verification of the translation.

Instruments to be translated – School Questionnaire

– Grade 4 Teacher Questionnaire

– Annex on Reading and Mathematics activities (commonly known as the Grade 4 Opportunity to Learn (OTL) Questionnaire)

– School Tracking Form – Teacher Tracking Form

– National Project Manager (NPM) Questionnaire, to be completed by the National Project Manager, with help from person(s) responsible for the primary school curriculum. Translation of this questionnaire was optional, depending on the individual who completed it for the participating country.

National languages

In countries with one predominant language which is used throughout the education system, only a single national version of the instruments was needed.

In multilingual countries, there was a need to develop national versions of the School, Teacher and OTL Questionnaires for each language used for instruction in the sampled schools, unless a more effective strategy could be applied. The general principle to identify the national languages was that teachers and school heads should complete the WEI-SPS questionnaires in a language of which they had perfect command, that is:

• either the language of instruction they usually use in their school; or

• a national language in which they are supposed to be proficient (because they had secondary or tertiary instruction in it, or because all teachers in the country or region need to sit advanced examinations in that language to obtain a teacher certification).

Paraguay had initially identified Spanish and Guarani as their national languages.

However, none of the 100 schools with Guarani as the main language of instruction fell into the sample group.

In Malaysia, there are two categories of primary schools: the national and the national-type schools. National schools use Bahasa Malaysia as the medium of instruction. National-

(21)

Common languages across WEI countries

A number of WEI countries have common official languages (i.e. Spanish for Argentina, Chile, Paraguay, Peru and Uruguay; Tamil for India and Sri Lanka).

“Standard” Spanish versions of the questionnaires and glossary were translated by the UNESCO Regional Office for Latin America and the Caribbean (OREALC). They were then reviewed by two UIS staff members for consistency with the original English source version. Each NPM could then apply their own national adaptations to this version in order to obtain the “customized” Spanish version for their country. This process was designed to reduce translation costs, while increasing both the quality and comparability of the final versions.

No standard version of the questionnaires in Tamil was produced as each country produced its own national version.

Single language versions

The recommended procedure entailed having each questionnaire translated from English to the target language by two independent translators. The two translations were then reconciled by a third person and reviewed for equivalence against the English source version.

The use of a glossary was strongly recommended to ensure the best possible adaptations to every country's specific terminology. The generic glossary that the UIS provided had been updated to include findings from the pilot study; nevertheless, it was not meant to be exhaustive for every national system.

Translation team Translators

The NPM was requested to only use translators who have professional experience in translating from English into the target language(s). These translators should have:

– An excellent knowledge of English;

– An excellent knowledge of the target language;

– Experience in their country and cultural context;

– If possible, some familiarity in translating survey materials;

– Familiarity with educational issues; and

– For the OTL: familiarity with specific terms, namely in mathematics (e.g. ‘cube’ or

‘chance’). When in doubt, they consulted with subject teachers when translating these terms.

If more than one person was involved in translating, it was particularly important that they liaise in order to ensure consistency of the translation within and across all questionnaires.

(22)

National research staff

To solve issues related to national adaptations, translators were instructed to work in close collaboration with the NPM or the national research staff member who was responsible for the development of the national version(s). This person was responsible, in particular, for the following tasks:

i. Review the glossary and adapt it to the national education system. This had to be repeated for every target language;

ii. Provide the translator(s) with all documentation and materials needed for the translation:

– questionnaires, tracking forms, data entry manual and other materials to be translated

– WEI-SPS guidelines for translators – glossary

– NAT forms.

iii. Send the translated materials to the UIS for international verification of equivalence against the source version;

iv. Incorporate in the materials possible edits which were suggested by the international verifier; and

v. Ensure that the questionnaires were reviewed and received final endorsement by the national authorities or National Committee before going to print.

National Committee

Involving a National Committee in the process of instrument review/endorsement was an important step for many reasons. First, the experts involved provided useful advice to improve the materials. Second, their prior endorsement was intended to help prevent possible conflicts or criticisms when the study results were released, in the event that any unexpected or serious concerns about the national education system were raised. Third, the Committee members could facilitate the dissemination of the study results by identifying the various constituencies that may be interested in particular issues and advising on the best way to convey the information collected to each of them.

Adaptation procedures

There are many linguistic, cultural and organizational differences across school systems, which needed to be accounted for to ensure that the data were collected in equivalent ways. Adaptations were needed in all national versions, including the English versions of the questionnaires used in the Philippines.

(23)

• Refrain from dropping questions, response categories and completion directives from the questionnaire, unless it could be clearly proven that keeping these would have negatively affected responses in the country;

• Abstain from adding notes or examples to questions unless it could be clearly proven that questions would not have been answered without them (however, this could affect international comparability);

• Limit the number of national questions as this could affect response rates;

• Keep additions separate from international questions; and

• Fully document adaptations, deletions and additions in the NAT forms.

It should be noted that India submitted an English version of its adapted questionnaires for review by the UIS before translating them into three other languages (Assamese, Hindi and Tamil).

3.2 Verification

National versions of the questionnaires were submitted by NPMs to the UIS for verification of:

i. the quality of the translation. Verifiers checked for possible mistranslations or lack of linguistic equivalence against the English source versions, included their suggested edits in the national version of the questionnaires and added notes about any undocumented deviations in the NATs provided by the NPMs; and ii. the appropriateness of national adaptations made by English-speaking UIS

researchers who used the NAT forms to check whether the national adaptations were acceptable, asked for additional explanations from the NPM or from the linguistic verifier in case of doubtful changes, and finally approved or disapproved each adaptation.

Depending on the language, the verification of the translation quality was performed either by a verifier from a linguistics quality control firm or a verifier at the UIS. Comments were reviewed by the UIS and then sent back to the NPMs for corrections or for further clarifications. Corrected national versions were then sent again to the UIS for the second verification step. This process was iterative until final approval was given by the UIS for printing all national documents.

Submission of translated material for central verification

All national versions used for the main survey were submitted in MS Word format to the UIS, together with a copy of the NAT forms for verification of their equivalence against the international source version of the instruments. Countries also submitted the glossary of terms they prepared to assist their translators.

All countries participating in the SPS data collection submitted national versions of instruments for translation verification.

(24)

Translation verification process

The appointed international verifiers had a native command of the target language and fluent command of English. They received training to help them perform their task, which consisted of:

• Verifying the linguistic correctness of the translated materials, their adherence to the WEI-SPS Translation Guidelines and their equivalence against the English source version;

• Checking whether all approved adaptations were consistently incorporated in the materials;

• Ensuring that no undocumented or unapproved adaptations were included; and

• Checking whether possible last-minute changes or errata circulated by the UIS/OECD were correctly implemented.

Verifiers were provided with general information about the study. They also received materials describing the translation procedures used by the national centres, cultural adaptations deemed acceptable, and detailed instructions for reviewing the instruments.

The verification guidelines emphasized the importance of maintaining the meaning and format of each question, while allowing for cultural adaptations as needed. Items in the OTL also had to maintain their difficulty level and could only be adapted with regards to cultural issues (e.g. “cake” instead of “pizza”).

Each verifier received:

– an international version of each questionnaire – national version of each questionnaire

– NAT forms

– SPS glossary with corresponding terms in the target language – translated School Tracking Form

– translated Teacher Tracking Form – guidelines for translation verification.

Possible edits suggested by the verifiers were entered in “track changes” mode in the electronic Word files provided by the NPM. This helped the national team perform the final revision of their materials (the NPM was able to “accept” or “reject” each particular correction proposed by the verifier, without having to create a new entry in the file).

When undocumented adaptations were found in the materials, the verifier was instructed to note them in the NAT form and return the form to the person in charge of adaptations approval at the UIS, who then informed the NPM on whether the new adaptations were approved or not.

Before the final approval of the documents, NPMs were asked to submit a PDF version of the documents for final review by the verifiers. This last review was principally done to

(25)

Three countries (Brazil, Paraguay and the Philippines) did not submit PDF versions, because the conversion from Word to PDF format caused visual inconsistencies in their documents. Therefore, verification of the layout was performed on the Word version and proved to conform to the English source documents. Nevertheless, approval to print was given to these three countries with the condition that their printed versions follow the same layout as the approved Word version. Finally, paper copies of the documents were provided to the UIS as proof that layout was consistent with the English version.

While it was expected that most of the edits suggested by the international verifier would be accepted, it is important to note that, in the case of a disagreement, the NPM retained the final responsibility for the quality of the materials used in the country. Disagreements and national adaptations are detailed in an Access database entitled "National Deviations Database". The records from the database are published in Appendix VI.

National Adaptation and Translation (NAT) forms NAT forms were used by:

• Countries for documenting translations and adaptations. They had to be submitted for verification, together with the questionnaires;

• Verifiers for registering all deviations and suggesting changes in each instrument; and

• The UIS to submit comments and final approval.

The usual processing of NAT forms can be described as the following iterative loop:

• The country submitted the completed NAT forms with the questionnaires to the UIS;

• The UIS verified the adaptations described in the NAT forms then either approved, disapproved or posed questions as to the intent of the verifier;

• The UIS submitted the NAT forms to the verifier;

• The verifier registered all deviations and suggested changes in the NAT forms. The verifier also answered the questions that the UIS included in the NAT forms;

• The verifier returned the NAT forms to the UIS;

• The UIS verified the NAT forms once again to approve, disapprove or pose questions to the country; and

• The UIS returned the NAT forms to the country for corrections or questions.

The iterative loop starts again from the beginning of the list. The loop continues until all corrections are made, a PDF version is reviewed, and final approval granted by the UIS to print the questionnaires.

The iterative loop of translation verification was performed for the main survey only. For the pilot, translation verification was done in one step – the country submitted their questionnaires and a verifier at the UIS reviewed the questionnaires without extensive use of the NAT forms. The main objective of the translation verification during the pilot phase

(26)

For the main survey, most questionnaires and NAT forms were verified using the iterative loop. The countries whose documents were verified iteratively until approval by the UIS are Argentina, Brazil, Chile, India (Hindi and Tamil), Paraguay, Peru, the Philippines, Sri Lanka (Sinhala and Tamil), Tunisia and Uruguay.

The countries whose documents were verified using a two-step method are India (Assamese) and Malaysia (Bahasa Malaysia). Before the start of the verification of the questionnaires, India and Malaysia submitted the glossary to the UIS for approval of the adaptations. For step one, the verifier performed the usual detailed verification and checked for consistency by comparison with the source version. Then, the documents were sent back to the country for corrections and they were asked to provide a PDF format of their questionnaires. For step two, the verifier performed the visual verification of the layout and checked whether major corrections identified in the first step were done. No major uncorrected errors were found by the verifiers in step two. India’s Assamese and Malaysia’s Bahasa Malaysia documents were not iteratively verified because the two-step method proved to be sufficient to produce an internationally comparable version.

Although most countries submitted their NAT forms together with the questionnaires, a few countries did not submit their NAT forms in time for the translation verification to be performed concurrently with that of their first national version of the questionnaires. These countries include: India (Assamese, Hindi and Tamil) and Sri Lanka (Sinhala and Tamil).

3.3 Conclusion

In their survey activities report, NPMs were asked to report the procedures used for the development of their national version(s). The reports received from the countries, along with the verifiers’ reports, revealed that there were no items lost due to translation errors or any other translation issues that might have affected the international comparability of the instruments.

The translation procedures proved to be indispensable in ensuring the high quality of the national versions of the instruments. Its intent was to guarantee that the instruments had been translated precisely and were kept comparable to the original versions. The procedures instituted for translation, adaptations and translation verification provided international comparability of the questionnaires used by the WEI-SPS countries. In addition, the exhaustive process of review and verification ensured the accuracy in the analysis and reporting of the data.

(27)

Chapter 4. Sampling design

The international sample design for WEI-SPS is described in this chapter. It includes a description of the procedures developed to ensure that the national samples: i) were selected in a manner consistent with the recommended probability sample design; ii) yielded accurate, weighted survey estimates for which sampling variances were calculable; and iii) were internationally comparable across participating countries.

Each participating country was required to use a probability sample representative of the target population. The UIS provided countries with a manual that described the sample design and selection procedures. It also provided countries with advice on how to best meet the WEI-SPS quality standards and how to develop and implement sound national sampling plans.

As a quality control measure, each participating country submitted to the UIS a set of completed forms that included the intention to participate in the study and provided key information pertaining to the sampling plan. These sampling forms were reviewed by the UIS as part of the approval process. The information requested in each sampling form is presented in Table 4.

Table 4. Sampling forms summary

Form Form name Information

1 Participation Confirmation of participation; planned date of main survey; school year;

age and birth date rules for entering primary school; grade structure of primary schools.

2 National Desired

Target Population Description of national ‘desired’ target population; identification of target grade; age and enrolment statistics; specification of extent of national coverage; description of exclusions.

3 National Defined

Target Population Description of national ‘defined’ target population; specification of school level and within-school exclusions.

4 Stratification Identification of explicit and implicit stratification variables.

5 Small Schools Pupil enrolment data regarding prevalence of small schools.

6 Sampling Frame Description of sampling frame; measure of size (MOS).

7 Excluded

Schools List and description of schools excluded from national defined target population; reason for exclusion; number of pupils in excluded schools.

8 Sampling Frame Statistics

Counts of the number of schools and eligible pupils for each stratum.

9 Sample

Allocation Size of sample allocated to each explicit stratum.

10 School Sample

Selection Sample selection information for each explicit stratum that illustrates how the school sample was selected.

11 Sample Frame (Sample IDs)

Written record of school sampling frame and school sample selection (preferably submitted as a computer file with all relevant information included).

12 School Tracking

Form List of selected schools to keep track of the participation status of all sampled schools and replacement schools.

Note: See Appendix V for copies of the sampling forms.

(28)

4.1 Target population

The WEI-SPS was designed to collect data pertaining to two types of survey units: schools and teachers. The defining characteristics of these two target populations are described in this section.

Schools

The WEI-SPS international target population of schools was defined as “all schools with pupils enrolled in the fourth grade”. A school was defined as an administrative unit or a school site. All schools of educational sub-systems with pupils in the fourth grade were included in the international target population. Schools that did not contain any pupils in Grade 4 were excluded from the study.

A country’s desired target population may have differed from the international definition for a number of reasons. Countries were permitted to reduce the SPS national coverage for political, organizational or operational grounds. For example, the removal of a geographical region, an educational sub-system or even a language group from the nationally-desired target population was acceptable.

A nationally-desired population may have been further refined to exclude schools due to:

• Geographic inaccessibility (e.g. schools in remote areas).

• Extremely small size, which is subject to the following guidelines:

– Countries were permitted to exclude extremely small schools provided the resulting exclusions were less than 2% of the total Grade 4 enrolments and the overall set of exclusions did not exceed 5% of the total Grade 4 enrolments.

– If enrolment in small schools was above 2% but represented less than 10% of all national Grade 4 enrolment, then small schools remained on the school sampling frame and were subject to normal sampling procedures.

– If the enrolment in small schools exceeded 10%, countries were requested to form an explicit stratum of small schools or use a double-level frame approach if operational costs were an issue.

• Exceptionally different curriculum or school structure.

• Provision of instruction only to pupils that were mentally or functionally disabled (e.g. schools for the blind).

The exclusion of small schools was permissible because of the relatively high cost of data collection compared to the small ratio of teachers to Grade 4 pupils. Each country was instructed to create its own definition of a small school. Thus, the definition of a small school varied from one country to another.

After permissible exclusions, the nationally-defined target population of schools was in place.

Each participating country was required to limit school exclusions to a maximum of 5% of the

(29)

A nationally-defined target population of schools was restricted to, and defined by, the schools included in the available sampling frame. Figure 2 illustrates the relationships among the levels of target populations and the possible reductions in coverage and exclusions. Ultimately, the national effective target population is represented by the sample of participating schools and teachers after all sources of exclusions have been taken into account.

Hence, the SPS weighted estimates are representative of the national effective target population of schools. The only within-sample exclusions were sampled schools that were deemed ineligible during data collection (i.e. schools with no pupils enrolled in the fourth grade).

Figure 2. Sample coverage and exclusions

Exclusions from National Coverage

School-Level Exclusions

Within-Sample Exclusions National Effective Target Population

National Defined Target Population National Desired Target Population

International Desired Target Population

Source: WEI-SPS Sampling Manual.

Teachers

The WEI-SPS international target population of teachers was defined as “all teachers, within the school target population, who were teaching <language> and/or mathematics/arithmetic to Grade 4 pupils”. The language taught refers to the main language of instruction and does not include a foreign language taught as a second language.

The school exclusions described in the previous section also would have resulted in the exclusion of teachers from these excluded schools. In addition, exclusions from the target population may have included within-school exclusions as follows:

• Teachers of classes only attended by mentally disabled pupils: These were pupils who were considered to be mentally disabled in the professional opinion of the school principal or other qualified staff members, or who had been determined to be mentally disabled as a result of a psychological test or evaluation.

• Teachers of classes only attended by functionally disabled pupils: These were pupils with a permanent physical disability that prevented them from undertaking school testing.

(30)

A country’s desired target population of teachers was restricted to, and defined by, the schools included in the available sampling frame, since only teachers within these schools were available for sampling. Individual teachers could not be excluded, hence the nationally-desired target population became the nationally-defined target population. Therefore, the SPS weighted teacher-based estimates are representative of this nationally-defined target population of teachers.

4.2 Sample design

As mentioned in Chapter 1, the objective of the study was to obtain cross-national data to satisfy information needs related to equity and quality issues in education. The WEI-SPS sampling objective was to use a sample design that would yield reliable survey estimates at the national level for each participating country.

A key sample design objective was to provide reliable pupil-based and school-based estimates for schools and teachers. Thus, there was a need for a sample of schools and a sample of teachers.

The general sample design strategy was determined at a meeting of experts organized by the UIS in January 2003. The document “Sampling Meeting-Final Summary, UIS-OECD-WEI- SCGSVY (2003)” summarized the main issues addressed at the meeting and the conclusions.

The document states: “Since both the school and student are units of analysis, it was recommended that countries should use a disproportionate stratified sampling design with schools of different sizes having different probabilities of selection. In addition to being simple and easy to implement by countries, this design has also the advantage of providing a compromise precision on both student- and school-based estimates. The Probability Proportional to Size (PPS) technique would have been proposed if the unit of analysis was generally to be the child”.

Schools

The WEI-SPS study employed a stratified systematic sample design. Countries used either a single-stage procedure whereby the sample of schools was selected directly from a list of eligible schools, or a two-stage procedure which first involved the selection of a number of Primary Sampling Units (PSUs), followed by the selection of the sample of schools from the list of eligible schools within the selected PSUs. For each defined stratum, schools were selected with equal probability using a systematic sampling technique.

Teachers

In each selected school, all teachers teaching the main language of Grade 4 instruction and/or mathematics/arithmetic to Grade 4 pupils were included in the national teacher sample.

(31)

Sampling precision and sample size

The WEI-SPS sampling precision required that each participating country achieve an effective sample size of at least 400 schools. This sample size ensured that the standard error of the SPS estimates would not be greater than those obtained from a simple random sample of 400 schools. In other words, final sample data were “effective” in achieving the prescribed precision if the country collected completed SPS questionnaires for at least 400 schools.

With an effective sample size of at least 400 schools, an approximate 95% confidence limit for sample estimates of population means and percentages can be expected:

• Means: m ± 0.1 s (where "m" is the mean estimate for schools and "s" is its estimated standard deviation);

• Percentages: p ± 5% (where "p" is a school percentage estimate).

The WEI-SPS study proposed the selection of schools with a single-stage procedure using a stratified sampling design, which was expected to be as efficient as a simple random sample.

For countries that used this recommended approach, the precision criteria would be achieved.

However, any country that used a two-stage procedure to select the sample of schools introduced a degree of sampling error due to the clustering of schools into first-stage sampling units. A two-stage procedure was appropriate when there was no comprehensive national list of schools or the costs to collect the data were high due to travel distances. In this case, a larger sample size was required to achieve the precision objective.

The sample of teachers was selected using a stratified multi-stage design. This design which involves the clustering of teachers by selected schools is generally less efficient than a simple random sample. Nevertheless, the selection of teachers from each selected school helped to minimize the data collection costs that would have been associated with a separate teacher sample, and it also enabled the linkage of the school data with the data for teachers and students. Furthermore, the inclusion of all teachers within a selected school resulted in a teacher sample that was considerably greater than 400, thus reducing the sampling error due to the cluster effect.

The student-based estimates were also derived from the stratified multi-stage design for teachers. Similar to the case with teachers, any increase in the sampling error of the student- based survey estimates due to the cluster sample is offset by a sample size that is much larger than the recommended 400 minimum sample size.

Stratification

Stratification was recommended as a key feature of the SPS sample design. It consists of grouping schools, and hence teachers, according to a characteristic or variable prior to the selection of the sample. In general, stratification is an important feature of a sample design for the following reasons:

• it improves the efficiency of the sample design, thus helping to minimize the standard error of the survey estimates;

• it ensures the representation of survey units by the variables considered important with respect to the survey estimation objectives;

(32)

target population members or schools within geographic areas (e.g. states or provinces);

• it facilitates the production of survey estimates using the explicit stratification variables;

and

• it enables the allocation of a sufficient sample to the explicit strata so that a country’s precision objective for stratum-level estimates may be possible.

Countries were encouraged to use explicit and implicit stratification in the WEI-SPS.

Explicit stratification

Explicit stratification entails building separate school lists or sampling frames according to a set of explicit characteristics or variables. For example, the explicit stratification of the target population of schools by urbanization required the grouping of schools based on location (urban or rural area). It is possible that different sample designs or simply a constant sample size could then be applied to each school sampling frame to select the sample.

Explicit stratification was recommended as a sample design feature for SPS. In particular, it would apply a disproportionate allocation of the school sample to the explicit strata. For example, in a country where geographic region was an explicit stratification variable, the same number of schools could have been sampled from each region regardless of the relative size of the regions. The regional stratification in this situation would satisfy the objective to produce equally reliable school estimates for each region.

The minimum requirement for SPS explicit stratification was the grouping of target population schools by school size (measured according to the number of Grade 4 pupils), since it could vary considerably within a country or geographic region. It was anticipated that school size would affect the variance of at least some of the SPS school-based estimates. Therefore, the stratification strategy by school size avoided the possibility of including widely varying school sizes within the same stratum, helping to minimize the sampling error of the school-based estimates. It was suggested that as many as five school-size strata be formed so that each stratum comprised approximately 20% of the total Grade 4 pupil enrolment.

Other suggested stratification variables included geographic regions (e.g. states or provinces), urbanization (rural or urban areas), socio-economic status (low, medium or high income), and school type (public or private). Countries could choose which variable to use to define explicit strata or alternatively to implicitly stratify the list of schools. A country with an objective to produce sub-national estimates within a specified margin of error defined explicit strata corresponding to the sub-national domains of interest. Thus, the country was able to allocate its sample to these strata so that its specified precision could be achieved for the desired domain estimates. Of course, each country had to achieve a balance between any objective for sub- national estimates and the cost of achieving such an objective. For example, if survey estimates were required by region, then a country was required to ensure that each region’s sample size was large enough to produce reasonably reliable regional estimates. In many cases, a goal to