8. Communications, uses and impact of large-scale assessments
8.2 Informing policymaking
informing education policy has mainly relied on two approaches. One is to collect data on a myriad of school and classroom factors and determine the relationships of these factors with learning outcomes.
The results of these analyses are used to support various national policies. The second approach is for countries to compare their results with those of other countries. The factors considered relevant to student success are grouped into a number of policy themes, which can be broadly categorised as school resources, accountability, school governance, teaching practices and selective schooling.
Both approaches are problematic and can fail to sufficiently address the cumulative result of countless factors that affect children’s development, beginning at conception and continuing. Moreover, the measures of the key school factors that do affect student performance tend to be inter-correlated and
Figure 8.2 School-level performance by average pupil background, India
strongly correlated with the average socioeconomic status of the school. It is virtually impossible to isolate the “school effects” attributable to particular resources or processes with a cross-sectional study.
In an attempt to understand the relationship between pupils’ home advantages and their school-level performance, Desa et al. (2008) examined learning outcomes in four main types of schools in India.
While there is considerable variation in both the mean school English score and mean pupil assets score, there is a strong general pattern demonstrating that more advantaged pupils attend higher performing (and mainly private, unaided) schools. Overall, there is greater variation in academic performance for schools attended, on average, by more disadvantaged students. This finding is partly a function of the type of school attended (state schools have more variation in performance), but it is notable that within private schools there is more variance regarding performance
than in schools attended by the most disadvantaged pupils. There is no discernible pattern among state schools, and there is no school type which, on average, has more advantaged pupils. In addition, a large proportion of state schools have lower performance than almost all private schools, due in part to their high concentration of disadvantaged students. In general, disadvantaged pupils attend lower-performing schools and schools which are less effective and also have greater variation regarding performance. Figure 8.2 illustrates the general relationship between students’ home advantage and their school-level performance.
The 2030 Agenda for Sustainable Development focuses on inequalities: gender, disability, immigrant status, language spoken at home and at school, and poverty, identified according to different databases.
20 40 60 80 100
Mean School English Score (%)
-2 0 2 4
Mean Student Wealth Index
Private aided Private unaided State school Tribal and Social Welfare schools Source: Crouch and Rolleston, 2017.
Figure 8.2 School-level performance by average pupil background, India
8.2.1 Simulating intervention effect
Monitoring data can inform policy questions about the performance of the school system and serve to monitor progress. This section considers three ways to frame questions about strategies and their execution.For each strategy, the potential effect of a hypothetical strategy is also considered. In Figures 8.3 and 8.4 displaying the hypothetical effects, the red gradient line displays the “before intervention” status, which is set to the gradient, and the green line displays the hypothetical “after intervention” gradient.
8.2.2 Universal versus targeted strategies
Figure 8.3 shows two different strategies, one universal and the other targeted at a certain level of performance. It summarises the learning bar in reading according to the socioeconomic status of the student. As expected, the level of learning increases with socioeconomic status as represented by the red line. The effect of simulation is shown by the green line.The figure at the top shows a universal strategy which strives to improve the outcomes of all students in a jurisdiction. Reform could take place through different means: curriculum reforms, reducing class size, changing the age of entry into kindergarten or increasing the time spent on reading instruction are all universal strategies as they are targeted towards all students, irrespective of socioeconomic status.
Consistently the bar in green simulates the effect of the universal policy.
The figure on the bottom shows a strategy that is targeted towards students with low levels of performance based on an outcome in a performance-targeted intervention. Using sampling information from large-scale assessments, governments could target a school level by the type of school. A performance-targeted strategy can also be implemented at the school level. For example, a reading programme
may be administered in a sample of schools that have low average performances. In school systems with a low vertical inclusion index, it is efficient to implement a whole-school strategy in a small number of schools. A vulnerability concentration plot can be used to estimate the number of children that would be reached with an intervention in a particular number of schools. The classification provides teachers with information regarding the type and amount of support required for each child.
8.2.3 Compensatory strategy
Figure 8.4 shows the effects of a compensatory strategy that raises socioeconomic status. A
compensatory strategy provides additional education resources to students from low socioeconomic backgrounds or students deemed “at risk” for other reasons. The term, “at risk” can refer to being at risk of not successfully achieving a particular development outcome or more generally at risk of poor
development for a range of developmental outcomes.
What is the difference between this intervention and the targeted one? The targeted sub-population can be the same as for an socioeconomic status-targeted intervention. A compensatory strategy intends to improve learning outcomes by improving the socioeconomic situation by using social and economic policy. This is a difference with respect to previous examples where the “compensation” comes from educational tools and policies as a way to improve outcomes. Providing free breakfast or lunch programmes or free textbooks for low socioeconomic status students are compensatory strategies.
However, the effect is difficult to measure and it is indirect and dependent on many other factors. So far, no marked effect on improving children’s outcomes has been proven.
Figure 8.3 The effect of a universal and a targeted policy
700
600
500
400
300
200
Reading proficiency
Student socioeconomic status
Figure 8.3 The effect of a universal and a targeted policy
Level 6 Level 5
Level 4
Level 3
Level 2
Level 1
< Level 1
-3 -2 -1 0 1 2 2
700
600
500
400
300
200
Reading proficiency
Student socioeconomic status
Level 6
Level 5 Level 4
Level 3
Level 2 Level 1
< Level 1
-3 -2 -1 0 1 2 2
Mexico
Note: The red bar represents the pre-intervention learning bar in reading according to socioeconomic status. The green bar represented the post-intervention learning bar.
Source: Willms, 2018.
Mexico
Universal policy
Targeted for low socioeconomic status students