Successful PISA Stories in the EU (2022) PDF

Successful PISA stories in the EU: how some Member States have been able to improve their performance over time Final Report EUROPEAN COMMISSION Directorate-General for Education, Youth, Sport and Culture Directorate A — Policy Strategy and Evaluation Unit A.2 — Country analysis E-mail: [email protected] European Commission B-1049 Brussels EUROPEAN COMMISSION Successful PISA stories in the EU: how some Member States have been able to improve their performance over time Final Report Directorate-General for Education, Youth, Sport and Culture 2022 Education and Training EN Disclaimer This document has been prepared for the European Commission. However, it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. Getting in touch with the EU Europe Direct is a service that answers your questions about the European Union. You can contact this service: by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), at the following standard number: +32 22999696 or by email via:https://europa.eu/european-union/contact_en Luxembourg: Publications Office of the European Union, 2022 © European Union, 2022 Reuse is authorised provided the source is acknowledged. The reuse policy of European Commission documents is regulated by Decision 2011/833/EU (OJ L 330, 14.12.2011, p. 39). For any use or reproduction of photos or other material that is not under the EU copyright, permission must be sought directly from the copyright holders. Image(s) © Item ID: 1767368444. Source: shutterstock.com PDF ISBN 978-92-76-49254-2 doi: 10.2766/429517 NC-06-22-160-EN-N The core research team: Hanna Siarova, Project Director (PPMI, Lithuania) Prof Dr Dominique Lafontaine, Key expert (University of Liege, Belgium) Prof Dr Ariane Baye, Key Expert (University of Liege, Belgium) Dr Valérie Quittre, Key expert (University of Liege, Belgium) Fabian Pressia, Key expert (University of Liege, Belgium) Radvilė Bankauskaitė, Researcher (PPMI, Lithuania) Katya Dunajeva, Senior Researcher (PPMI, Hungary) National experts: Dr Kiril Bankov (Bulgaria) Gunda Tire (Estonia) Dr Andris Kangro (Latvia) Dr Grzegorz Mazurkiewicz (Poland) Dr Gil Nata (Portugal) Klaudija Šterman Ivančič (Slovenia) Dr Urška Štremfel (Slovenia) Dr Samuel Sollerman (Sweden) Maria Nordlund (Sweden) Acknowledgements: The research team would like to thank national experts and stakeholders from the focus countries (Bulgaria, Estonia, Latvia, Poland, Portugal, Slovenia, and Sweden) for supporting the data collection. We also gratefully acknowledge the advice and useful comments from the advisory experts, whose feedback informed the methodological approach and analysis. The team is also grateful to the Steering Committee of the project (DG EAC) for their support, guidance, and feedback throughout the study. Contractor: Public Policy and Management Institute Gedimino pr. 50, LT-01110 Vilnius, Lithuania Phone: +370 5 249 7056 http://www.ppmi.lt Director: Mr Rimantas Dumčius PISA success stories Table of contents Table of contents............................................................................................... 5 List of figures.................................................................................................... 7 List of tables..................................................................................................... 8 List of boxes..................................................................................................... 8 Executive summary..........................................................................................10 Methodological approach....................................................................................10 Key findings of the study....................................................................................12 Factors associated with improvements in PISA performance.................................12 What policies are associated with student success: effective and promising reforms?....................................................................................................................13 Lessons for policy design and implementation.....................................................17 Introduction.....................................................................................................21 1.1. How to read this report.............................................................................22 1.2. Aims, objectives and research questions.....................................................22 1.2.1. Specific objectives.................................................................................23 1.2.2. Study research questions.......................................................................23 1.3. Overview of methodological approach.........................................................24 1.3.1. Scope of the study................................................................................24 1.3.2. Selection of PISA success stories.............................................................24 1.3.3. Research process..................................................................................27 1.3.4. Data and measurements........................................................................28 1.3.5. Study limitations...................................................................................28 2. Overview of the EU’s performance in PISA between 2009 and 2018................30 2.1. General trends across the EU.....................................................................30 2.2. Zooming in on seven countries...................................................................33 2.2.1. Overview of trends in certain variables among the selected countries..........34 2.2.2. Factors influencing students’ performance in the selected countries.............40 3. Country PISA stories.................................................................................46 3.1. Bulgaria..................................................................................................46 3.1.1. Educational context and key policy issues.................................................46 3.1.2. Key trends in students’ performance........................................................47 3.1.3. Key policies associated with the improvements observed............................51 3.1.4. Lessons for future education reforms in Bulgaria.......................................62 3.2. Estonia...................................................................................................63 3.2.1. Educational context and key policy issues.................................................63 3.2.2. Key trends in students’ performance........................................................64 3.2.3. Key policies associated with the observed improvements............................68 3.2.4. Lessons for future education reforms in Estonia........................................80 3.3. Latvia.....................................................................................................81 3.3.1. Educational context and key policy issues.................................................81 3.3.2. Key trends in students’ performance........................................................82 3.3.3. Key policies associated with the improvements observed............................86 3.3.4. Lessons for future education reforms in Latvia..........................................95 5 June 2022 PISA success stories 3.4. Poland....................................................................................................96 3.4.1. Educational context and key policy issues.................................................96 3.4.2. Key trends in students’ performance........................................................97 3.4.3. Key policies associated with the improvements observed.......................... 100 3.4.4. Lessons for future education reforms in Poland....................................... 109 3.5. Portugal................................................................................................ 109 3.5.1. Educational context and key policy issues............................................... 109 3.5.2. Key trends in students’ performance...................................................... 111 3.5.3. Key policies associated with the improvements observed.......................... 116 3.5.4. Lessons for future education reforms in Portugal..................................... 133 3.6. Slovenia................................................................................................ 134 3.6.1. Educational context and key policy issues............................................... 134 3.6.2. Key trends in students’ performance...................................................... 135 3.6.3. Key policies associated with the improvements observed.......................... 139 3.6.4. Lessons for future education reforms in Slovenia..................................... 148 3.7. Sweden................................................................................................. 149 3.7.1. Educational context and key policy issues............................................... 149 3.7.2. Key trends in students’ performance...................................................... 151 3.7.3. Key policies associated with the improvements observed.......................... 155 3.7.4. Lessons for future education reforms in Sweden...................................... 172 4. What can we learn from the country reforms?............................................ 174 4.1. What interventions can help to improve the quality and inclusiveness of education systems?........................................................................................................ 181 4.1.1. Structural changes to the education system............................................ 181 4.1.2. Curriculum and assessment reforms...................................................... 183 4.1.3. Interventions targeting the equity of education provision......................... 184 4.1.4. Strengthening monitoring and evidence-based decision-making................ 185 4.2. Lessons for policy design and implementation............................................ 186 4.2.1. Context-sensitivity.............................................................................. 187 4.2.2. Inclusive policy design......................................................................... 188 4.2.3. Adequate time for planning and implementation...................................... 189 4.2.4. Coordination of reforms....................................................................... 191 4.2.5. Ensuring an adequate delivery system................................................... 193 5. Conclusions and implications for future policy and research......................... 198 5.1.1. Key findings of the study: finding a successful PISA story is challenging..... 198 5.1.2. Factors associated with improvements in PISA performance..................... 199 5.1.3. Effective and promising reforms: conclusive evidence is not yet there........ 200 5.2. Lessons for policy design and implementation............................................ 202 5.3. Key learnings from the study for future research........................................ 205 References..................................................................................................... 208 6 June 2022 PISA success stories List of figures Figure 1. Mean performance in all three domains in selected countries in 2018.................. 35 Figure 2. Trends in the selected countries in relation to selected variables........................... 38 Figure 3. Mean PISA performance and underachievement rates in Bulgaria 2000-2018... 48 Figure 4. Mathematics performance gaps between students in Bulgaria 2009-2018.......... 49 Figure 5. Variation in social and academic segregation in Bulgaria 2009-2018................... 50 Figure 6. Findings of the multi-level regression analysis of PISA 2018 data for Bulgaria.. 51 Figure 7. Reforms potentially linked to improvements in academic achievement in Bulgaria....... 52 Figure 8. Success factors and challenges affecting the effectiveness of the introduction of mandatory pre-school education (2002)........................................................................................... 56 Figure 9. Success factors and challenges affecting the effectiveness of the introduction of Matriculation exams (2003) and standardised national assessments (2007)........................ 60 Figure 10. Mean PISA performance and underachievement rates in Estonia 2000-2018.. 65 Figure 11. Mathematics performance gaps between students in Estonia 2009-2018......... 66 Figure 12. Variation in social and academic segregation in Estonia 2009-2018................... 67 Figure 13. Findings of the multi-level regression analysis of PISA 20018 data for Estonia.... 68 Figure 14. Reforms potentially linked to improvements in Academic Achievement in Estonia.... 69 Figure 15. Success factors and challenges affecting the effectiveness of the assessment system in Estonia..................................................................................................................................... 72 Figure 16. Success factors and challenges that affected the curricular changes in Estonia since 1996.................................................................................................................................................. 76 Figure 17. Main success factors and challenges influencing the effectiveness of the Newly introduced Rajaleidja centres............................................................................................................... 80 Figure 18. Mean PISA performance and underachievement rates in Latvia 2000-2018.... 83 Figure 19. performance gap between students in Mathematics in Latvia 2009-2018......... 84 Figure 20. Variation in school segregation in Latvia 2009-2018............................................... 85 Figure 21. Findings of the multi-level regression analysis of PISA 2018 data for Latvia... 86 Figure 22. Reforms potentially linked to improvement in academic performance in Latvia....... 87 Figure 23. Main Enabling factors for, and challenges to, the implementation of bilingual education reform in Latvia (2004)...................................................................................................... 89 Figure 24. Success factors and challenges affecting the effectiveness of the curriculum change (2006) and the ESF project ‘Natural sciences and mathematics’ (2009-2011)..... 94 Figure 25. Mean PISA performance and underachievement rates in Poland 2000-2018... 97 Figure 26. performance gaps between students in Mathematics in Poland, 2009-2018.... 98 Figure 27. Variation in social and academic segregation in Poland, 2009-2018................... 99 Figure 28. Findings of the multi-level regression analysis of PISA 2018 data for Poland 100 Figure 29. Reforms potentially linked to improvements in academic achievement in poland. 101 Figure 30. Success factors and challenges affecting the effectiveness of the 1999 education reform........................................................................................................................................................ 107 Figure 31. mean PISA performance and underachievement rates in Portugal 2000-2018..................................................................................................................................................................... 112 Figure 32. performance gaps between students in Mathematics in Portugal 2009-2018 114 Figure 33. Variation in social, ethnic and academic segregation in Portugal 2009-2018. 115 Figure 34. Findings of the multi-level regression analysis of PISA 2018 data for Portugal...... 116 Figure 35. Reforms potentially linked to improvements in academic achievement in Portugal. 117 Figure 36. Success factors and challenges Affecting the effectiveness of the reorganisation of the school network............................................................................................................................ 120 Figure 37. Success factors and challenges influencing the effectiveness of high-stakes exams (2004/2005)............................................................................................................................... 124 7 June 2022 PISA success stories Figure 38. Success factors and challenges influencing the effectiveness of the TEIP programme and PNPSE programmes............................................................................................... 131 Figure 39. mean PISA performance and underachievement rates in Slovenia 2000-2018..................................................................................................................................................................... 136 Figure 40. performance gaps between students in Mathematics in Slovenia 2009-2018 137 Figure 41. Variation in social, ethnic and academic segregation in Slovenia 2009-2018 138 Figure 42. Findings of the multi-level regression analysis of PISA 2018 data for Slovenia..... 139 Figure 43. Reforms potentially linked to improvements in academic achievement in Slovenia. 139 Figure 44. Success factors and challenges influencing the effectiveness of the 1996 education reform in Slovenia.............................................................................................................. 142 Figure 45. Success factors and challenges influencing the effectiveness of VET reform.. 147 Figure 46. MEan PISA performance and underachievement rates in Sweden 2000-2018152 Figure 47. performance gaps between students in Mathematics in Sweden 2009-2018. 153 Figure 48. Variation in social, ethnic and academic segregation in Sweden, 2009-2018 154 Figure 49. Findings of the multi-level regression analysis of PISA 2018 data for Sweden..................................................................................................................................................................... 155 Figure 50. Reforms potentially linked to improvements in academic achievement in Sweden... 156 Figure 51. Share of students from independent schools in PISA............................................. 159 Figure 52. performance gap in Mathematics between students in independent and public schools....................................................................................................................................................... 161 Figure 53. gap in socio-economic status between students in independent and public schools..................................................................................................................................................................... 162 Figure 54. Success factors and challenges influencing the effectiveness of the reforms introducing more support for independent schools...................................................................... 163 Figure 55. Success factors and challenges influencing the effectiveness of the changes in assessment and grading systems..................................................................................................... 169 List of tables Table 1. General research questions.......................................................................23 Table 2. Explanation of country selection.................................................................26 Table 3. Changes in PISA mean scores between 2009 and 2018.................................30 Table 4. Change in underachievement rate in EU Member States between 2009 and 2018..........................................................................................................................31 Table 5. Changes in some equity indicators in the EU between 2009 and 2018*...........32 Table 6. Main variables..........................................................................................34 Table 7. Variables included in the multi-level regression model..................................40 Table 8. Results from the multi-level regression on the seven selected countries..........41 Table 9. Overview of the reforms analysed in the report.......................................... 175 List of boxes Box 1. Overview of the introduction of one-year compulsory pre-school education (2002)..........................................................................................................................53 Box 2. Overview of the introduction of matriculation exams (2003) and standardised national assessments (2007).................................................................................58 Box 3. Changes in standardised national Assessment practices in Estonia....................70 Box 4. National curriculum reforms in Estonia since 1992..........................................73 Box 5. Introduction of Rajaleidja centres providing educational support to students with SEN....................................................................................................................78 Box 6. Introduction of Latvian as a language of instruction in minority schools.............87 Box 7. Curricular changes in Latvia in 2006.............................................................91 Box 8. ESF Project ‘Natural Sciences and mathematics’ for grades 7-9........................92 8 June 2022 PISA success stories Box 9. Changes in the structure of the Polish education system introduced by the 1999 reform.............................................................................................................. 102 Box 10. Curriculum changes introduced in 1999..................................................... 103 Box 11. Curriculum changes (2008)...................................................................... 103 Box 12. Changes in the national assessment system introduced by the 1999 education reform.............................................................................................................. 104 Box 13. School Network Reorganisation................................................................. 117 Box 14. Introduction of high-stakes national assessments in 2004/2005................... 122 Box 15. The TEIP programme............................................................................... 125 Box 16. the PNPSE programme............................................................................ 126 Box 17. reform of the structure of the Education system in Slovenia in 1996............. 140 Box 18. Changes in Slovenia’s VET system since 2006............................................ 145 Box 19. Decentralisation and Increased support for independent schools................... 157 Box 20. Changes to the standardised national assessment system in Sweden (2010-2012)........................................................................................................................ 165 Box 21. Changes in the grading system in Sweden................................................. 167 9 June 2022 PISA success stories Executive summary Mastering basic skills is a prerequisite for thriving in life, finding a fulfilling job and becoming an engaged citizen. At a micro-level, education plays an important part in determining social participation, well-being and employability; at a macro level, it is associated with higher levels of productivity and social cohesion. The development of the knowledge and technological society and the growing demand for the 21st-century skills in the labour market only increases the importance of education and the acquisition of foundational skills. However, one in five 15-year-olds in the EU fail to complete basic mathematics, science and reading tasks, according to the OECD’s Programme for International Student Assessment (PISA) (i.e., they do not achieve level 2 knowledge in the PISA assessment). Furthermore, students’ performance has gradually deteriorated over the period from 2009 to 2018. The cost of underperformance in education is rising. The COVID-19 pandemic, which caused school closures and learning disruption, may have further exacerbated educational inequalities, and negatively affected learning achievements in the long term. In such a context, it is imperative to understand which education reforms and interventions ‘work’ and thus lead to actual improvements in the quality and equity of education and the subsequent academic success of all students, in order to be able to maximise the returns from educational investments planned to support the recovery from the crisis and ensure just, green, and digital transitions, as highlighted by Strategic framework for European cooperation in education and training towards the European Education Area and beyond (2021-2030). To contribute to this understanding, the present study aims to explore which countries have been able to improve their students’ performance – looking at the PISA indicators measuring student achievement and equity – over time. It explores which factors can be associated with the positive trends observed, as well as which policy reforms may have contributed to these improvements. Methodological approach The study explores performance trends over the period from 2009 to 2018 (the last four PISA cycles), and education policy factors that might be associated with the positive trends observed. This is achieved by testing various statistical approaches to measure the effects of selected reforms (specifically, multi-level regression models, difference-in-difference and propensity score matching methods), complementing the analysis with insights from iterative stakeholder consultations conducted during the study, as well as the results of previously conducted non-experimental quantitative research. For the analysis, seven EU countries (the so-called PISA ‘success’ stories) – Bulgaria, Estonia, Latvia, Poland, Portugal, Slovenia, Sweden – were selected. Each of these countries has demonstrated, at least to some extent, long-term improvements in either performance or equity indicators, as measured by PISA1. 1 Further details about the methodology are available in Annex 2. 10 June 2022 PISA success stories Defining a PISA ‘success story’, and the challenges associated with doing so While students’ educational success is usually understood in a broader sense, for the purposes of this study it mostly refers to improved academic achievement in three subject domains (reading, maths and science) as measured by PISA, accompanied by a narrowing of the gaps in academic performance among students from diverse backgrounds, based on gender, migrant background and socio-economic status (SES). A country can be classified as a PISA success story if it managed to improve its students’ performance in at least one of the domains over time (from 2009 to 2018) while decreasing its rate of underachievement, and improving the equity of its education provision (i.e. the SES gap among achievers either decreased or remained stable). In terms of long-term changes (2009-2018) across the EU, several Member States have managed to improve their results in at least one domain since 2009. However, only in a few countries has this increase been statistically significant. The changes in underachievement rates over time in individual EU Member States mirror their PISA achievement scores, with only a few statistically significant improvements being observed. In most cases the observed improvements were minimal. Remarkable stability in academic achievement across the EU has been accompanied by persistent trends of inequality and exclusion in education systems. Equity indicators have improved unevenly among the Member States over the last decade. While more countries have managed to narrow the gender performance gap, significant differences in academic success still remain that depend on students’ socio-economic and cultural background. In this light, explaining a ‘PISA success story’ is, by definition, challenging when the success (i.e. observed improvement) is rather limited. Sweden Improved mean performance and reduced underachievement in maths, science and reading between 2012 and 2018 Estonia Improved mean performance and reduced underachievement in maths, science and reading between 2009 and 2018 Latvia Improved mean performance and reduced underachievement in maths between 2009 and 2018 Poland Improved mean performance and reduced underachievement in maths, science and reading between 2006 and 2018 Portugal Improved mean performance and Slovenia reduced underachievement in Improved mean performance and maths, science and Reading reduced underachievement in Bulgaria between 2006 and 2018 maths and science between Improved mean performance and 2009 and 2015, Reading decreased underachievement in between 2012 and 2015 math, science and reading between 2006 and 2012, but has worsened since then 11 June 2022 PISA success stories Key findings of the study Factors associated with improvements in PISA performance The statistical analyses (descriptive statistics and multivariate regression modelling) carried out in this study have confirmed the findings from the broader literature on the key determinants of students’ academic success (as measured by PISA). These are as follows: Individual and family background: gender, socio-economic status of the family, ethnic, cultural, linguistic and migrant background – explain a great share of differences in students’ performance. Academic path of students is associated with performance results. These include academic abilities of the students, the lack of which may result in following a vocational rather than an academic educational track, and grade repetition. School or instructional factors: multi-level regression run on the seven selected countries shows that several school-level variables were important for the academic performance of students. More specifically, having more than four periods of mathematics per week had a significant positive effect on students’ performance in mathematics in Estonia, Latvia and Portugal. Moreover, in every country except Portugal, the disciplinary climate variable at either the individual or school level (or both) had an effect on students’ academic performance. Lastly, perceived cooperation between students at school level influenced the academic performance of students in Bulgaria, Estonia, Slovenia and Sweden. In addition, Multi-level regression analyses conducted for individual countries highlighted the importance of support from teachers (in Sweden), interest in the course from teachers (in Sweden), lack of peer victimisation (in Portugal and Slovenia), and higher sense of school belonging (in Estonia). Note: Most of these factors, especially the background factors, are not amenable to change, but their effect can be alleviated by educational policies. Other factors that might seem influential at first sight were in fact reciprocal relationships, such as the disciplinary climate or sense of belonging: it is impossible in a cross-sectional design to rule out which is the cause, and which the consequence. 12 June 2022 PISA success stories What policies are associated with student success: effective and promising reforms? Analysis of PISA journeys and the accompanying reform processes in Bulgaria, Estonia, Latvia, Poland, Portugal, Slovenia and Sweden demonstrates that policy-makers have made, and are still making, significant attempts in their quest to improve educational outcomes and to address school-level inequalities by implementing various sets of reforms. In general, such reforms target structural changes, the development and assessment of competences, and the improvement of equity. Rigorously assessing the extent to which particular reforms and interventions ‘work’ and achieve their intended outcomes proves challenging, however, due to: ▪ Lack of an evidence-based education policy tradition in Europe, shaping specificities of reform design (reforms rarely include built-in monitoring and evidence collection mechanisms based on experimental research into the reform process). ▪ Limitations of available cross-sectional data on student achievements (most international and national student assessments), analysis of which does not allow causal inferences to be made with regard to specific policy impacts. Consequently, isolating the impact of educational policy reforms on PISA performance through statistical modelling is a challenging undertaking. ▪ Limitations around PISA data and measurements. Analysing data coming from different PISA cycles can prove challenging, due to changes in the methodologies used, available variables, and the computation process of various indicators between cycles. The main drawback of PISA and similar assessment surveys is the cross-sectional nature of their design. The evaluation of policies and reforms aimed at improving student’s performance requires longitudinal data, with the targeted individuals being observed both before and after such intervention. In contrast to this, each cross-sectional assessment is carried out on a different set of individuals, leading to a lack of essential individual data either before or after a policy or reform. While methodological advances in recent years have made it possible to use cross- sectional data sources, limitations remain regarding the possible depth of any analysis that can be conducted across PISA cycles. To overcome these constraints, at least to some extent, a mixed approach was applied: ▪ Sophisticated statistical approaches were used to measure the effects of specific reforms (namely difference-in-difference and propensity score matching methods). These methods also have their limitations, and require certain data conditions to be met, which are not always in place (only in three countries were the necessary conditions in place – in Bulgaria, Poland and Portugal). ▪ Iterative stakeholder consultations were also held to contextualise the quantitative findings and to better understand the process of reform implementation. Structural changes to education systems Given the various ways in which rigid school structures shape educational trajectories and consolidate educational inequality, some countries implemented fundamental reforms to change the structure and organisation of their education systems. These include extending the length of compulsory education (e.g., Poland, Slovenia); investing in early childhood education and care (Bulgaria); re-organisation of the school network (Portugal); as well as diversification of educational pathways (Slovenia, Sweden). 13 June 2022 PISA success stories The potential and effectiveness of structural reforms The introduction of compulsory pre-school for 6-year-olds in Bulgaria targeted vulnerable children at least to some extent, and was accompanied by an additional set of measures (such as professional development, ECEC curricula). The Difference-in-difference (DiD) regression model showed that the introduction of a compulsory year of pre-school education had a positive effect on students who attended ECEC, as a direct result of the reform. In addition, the statistical analysis showed that the reform has, to some extent, succeeded in targeting socio-economically disadvantaged students, though it has not yet ensured access to ECEC for all vulnerable students. However, the effectiveness of the reform was hindered by its rushed implementation and a lack of time to address infrastructure shortages and the financial costs associated with enabling the ECEC attendance of poorer families. These challenges are being addressed with the new re-iteration of the ECEC reform introducing mandatory pre-school education for children from 4 years old (adopted in 2020, coming into force in 2023/2024). The education reform in 1999 in Poland focused on several important aspects of the education system. The main changes to the structure of the system were a move away from eight-year basic education provision to a new division separating primary (six years) and lower secondary education (three years) levels. The delaying of streaming and the extension of compulsory education are considered to be the most important and most beneficial measures introduced by 1999 reform. It is also acknowledged that other changes, including the curriculum reform and the change in the national assessment system, have potentially contributed to gradual improvement in the quality of education and the academic achievements of students. Due to the changing demographic situation in the country and the allegedly inefficient management of its school system, a measure aimed at improving the school network (by closing small underperforming schools and introducing school clusters) was introduced in Portugal in 2002. According to stakeholders, the reform has potentially contributed to a reduction in school segregation and increasing academic performance among Portuguese students between 2006 and 2009, through the optimised use of resources, the increased accountability of schools, and the provision of more structured educational pathways for students. However, there is a lack of rigorous quantitative evidence regarding the effectiveness of the reform. Sweden began a large school decentralisation reform in 1992, transferring decision-making powers over the allocation of funds to municipalities, and implementing a large-scale school voucher programme that led to the development of publicly funded but privately operated schools. The reforms aimed to improve the management of the education system, to disrupt monopoly the public schools had, to provide more choice for parents and students, and to create an environment for innovation. Several studies have tried to assess the effect of the reform on school segregation and students’ outcomes, finding moderate effects on segregation and limited effects on students. However, the positive effect of this reform is not visible from the analysis of PISA data. Analysis points to a negative effect on students attending voucher schools compared to municipal schools at secondary level, with the effects being more negative for low-ability students. These results show that in certain contexts, school autonomy combined with school choice can have adverse effects on students’ performance and equity. Curricular reforms Competence development is targeted mainly through curricular reforms and accompanying assessment policies. The curricular reforms reviewed in this study (Estonia, Latvia, Slovenia and Poland) included some similar patterns, such as an emphasis on well- being, learner agency, the ability to solve problems and to navigate an uncertain world, accompanied by school and teacher autonomy in enacting and mediating the policy changes. Greater school autonomy has been proven to be associated with improved student achievement (Hanushek, Link, & Woessmann, 2013). However, decentralisation is only effective when there are educational indicators that guide teachers, schools and municipalities on how well the curricula is being implemented. This 14 June 2022 PISA success stories makes it imperative to review and align the whole student assessment framework, to ensure coherence around the curriculum reform. However, as practice shows, not all countries managed to adapt their national assessment frameworks to the new visions of competence and skills development (as observed in the cases of Bulgaria and Latvia). The potential and effectiveness of curricular reforms Estonia’s national curriculum, and the high level of autonomy it provides to schools, is seen as important enabler of Estonia’s PISA success. The national curriculum reforms (which have been taking place since 1992) are deemed to be successful, as they introduced modern education concepts early on and have followed a similar discourse throughout subsequent curricular changes. Existing studies on Estonia’s performance in PISA list the curriculum reform of 1996, which introduced competences into the curriculum and provided greater academic autonomy to schools, as an important reform that has potentially contributed to the high academic achievements of Estonian students (Tire, 2021). Furthermore, as PISA studies focus on knowledge application and higher-order thinking skills, reforms introducing competence- based curricula are also likely to improve student achievements in PISA, if they are implemented effectively and are translated into classroom practices. The VET reform introduced in Slovenia in 2006 granted greater autonomy to schools and teachers, allowing them to tailor the curriculum to the needs and interests of their students and the needs of the local labour market. The curriculum also introduced more academically oriented content, along with a focus on practical skills. According to stakeholder perceptions, the changes to the curriculum are potentially connected to improvements in the quality of the VET education system, and a decreasing performance gap between students in the VET and academic tracks. Existing studies also link greater autonomy with improved performance (Hanushek, Link, & Woessmann, 2013); however, more rigorous research is needed in specific context of Slovenia. An increase in mean PISA scores for Bulgaria in all three domains shortly after the introduction of external examination is sometimes associated with the potential success of the country’s assessment reform. Stakeholders link it to the increased accountability of schools and teachers, as well as the improved evidence base for education policymaking. The plausible effect of this reform could be inferred from the available scientific literature, in particular the work of Bergbauer et al. (2018). However, while some argue that assessments work as a tool for accountability, others argue that there is little evidence that high-stakes assessments can lead to educational improvements. It is argued that instead, assessments may even be harmful to students, and divert the focus of teaching away from holistic student development to teaching ‘for the test’ (Koretz, 2017). Furthermore, evidence from Bulgaria indicates that while there is increased availability of assessment data, some question the quality and rigour of the assessments introduced, and consequently the suitability of the data, as well as the capacity of policy-makers to use it for decision-making. Policies focusing on equity and quality The policies captured by this study focus on specific set of measures aimed at improving equity: delaying streaming (in Poland); introducing support networks for schools (Estonia); priority education policies and preventing grade repetition (Portugal); and addressing linguistic school segregation (Latvia). Evidence is inconclusive as to the effect of such interventions, due to limited monitoring of their implementation; however, insights from this study and the broader literature suggest that the impact of targeted measures (such as the TEIP programme in Portugal) can be mixed in terms of students’ outcomes and equity. Furthermore, such measures can be highly politicised and controversial (as the Latvian example suggests), and need to be adequately planned and resourced. 15 June 2022 PISA success stories The potential and effectiveness of interventions aimed at improving equity Several measures have been introduced over the years to ensure equity in the Portuguese education system. The most important changes include the introduction of the TEIP programme (zones of educational priority), especially the second phase of the programme, which has been running since 2006, as well as the introduction in 2016 of the PNPSE programme (national programme to promote educational success). The introduction of the TEIP programme may potentially be connected with an increase in Portugal’s PISA performance between 2006 and 2009; however, existing studies do not find this effect when comparing TEIP schools with schools outside the programme (Ferraz, Neves, & Nata, 2019). In relation to the PNPSE programme, the statistical analysis shows that not repeating a grade contributed to a large increase (more than 50 points) in the reading performance of students who would otherwise have repeated a grade under the conditions that were previously in place in 2009. Because students from programme schools are not distinguished from children attending other schools in PISA dataset, it is impossible to attribute this effect specifically to the PNPSE programme. Even so, it is likely that the existence of the programme contributed to the observed effect. These programmes may have been effective due to their focus on increased school autonomy, the increased capacity of participating schools and of local actors to address local challenges, and complementarity between the programmes. However, the effect of the programmes may be hindered by a lack of transparency and objectivity and the potential stigmatisation of the schools taking part in the aforementioned programmes. In Latvia, gradual transitioning to Latvian as the main language of instruction in ethnic minority schools has had mixed effects on student performance. While the initial flexibility offered to schools in choosing models of bilingual education appeared to positively affect the academic performance of Russian-speaking students (as measured by PISA), the requirement to take final exams in Latvian has negatively impacted exam results. The subsequent reinforcement of Latvian instruction has not yielded achievement gains in ethnic minority students (Ivlevs & King, 2014). The implementation of this reform was hindered by strong resistance from the public (due to insufficient public buy-in and consultations), as well as limited support for the professional community in ethnic minority schools to implement such a shift (lack of necessary teaching materials, professional development and the language proficiency of educators). Minority schools continue to be formally separate from Latvian schools. These reforms were often accompanied by an increased awareness of the need for strengthening education monitoring and national assessment frameworks to enable further educational improvements and evidence-based reforms. However, while the movement towards the use of evidence in education in Europe appears to be gradually asserting itself through political intent and initiative, it is yet to be systematically incorporated into the practices of the various Member States. To date, a large body of robust evidence comes from the US and UK, but less from European countries. The practice of ‘evidence-based education’ is rare. 16 June 2022 PISA success stories Lessons for policy design and implementation Even though the available evidence does not allow causal inferences to be made regarding the impact of most of these reforms on student outcomes (as measured by PISA), the stakeholder consultations and qualitative analysis conducted for this study bring valuable insights into the way in which reforms in the selected countries are initiated and translated into practice, which can either facilitate or impede the achievement of the intended outcomes. The study finds that reforms are likely to be perceived as being more effective and translate into practice more smoothly when: → they are implemented as part of a comprehensive complementary set of measures. Curriculum and assessment reforms in Estonia and Sweden demonstrate that timely support and capacity development for education professionals facilitated policy design and implementation on the ground. What can be also observed from the countries studied is that curriculum reforms are demanding in terms of their implementation, since they require changes in many aspects that might challenge the existing beliefs and subjective realities deeply embedded into the individual and organisational context. Factors such as cost, the uncertainty of the outcomes, the risk aversion of stakeholders (which triggered the reversal of the curricular reform in Poland) also create additional obstacles to initiating and materialising changes in the curriculum. Buy-in, and the active engagement of teachers and schools as enactors and mediators of the reform, as well as system-level monitoring to support the national education strategy (as shown by the Estonian example) are also important for the reform success. Conversely, ECEC reform in Bulgaria and language reform in Latvia show how the lack of an appropriate delivery system (such as funding, infrastructure shortages, or lack of teacher support) can impede the smooth implementation of an educational intervention. Pointers for policy It is important to understand and commit to the complex follow-through of reform priorities. This depends on the right sequencing and the existence of appropriate institutions capable of supporting the reforms from decision to implementation. Capacity bottlenecks tend to constrain the ability of policy-makers to implement reforms effectively and to reap the full benefits of specific policy initiatives. The need to address such constraints when implementing reforms should be a priority. Policies are not designed in a vacuum and must be articulated within an existing policy framework. Sometimes a reform contradicts or competes with existing policies and may create obstacles to effective implementation. For instance, increasing teachers’ workload or forcing teachers to invest in one policy at the expense of the other, are potential barriers to curriculum implementation. → they are well-planned and consistent with long-term education objectives. The importance of leaving adequate preparation time is illustrated by the 1996 education structural reform in Slovenia. While the changes to the education system were announced in 1996, they were implemented between 1999 and 2003, leaving time for schools, teachers and other relevant stakeholders to prepare for the changes. The case 17 June 2022 PISA success stories of Estonia illustrates the importance of contextualising new educational reforms within the existing landscape of educational policies, creating synergies and mutually strengthening the effects of these policies, and, importantly, ensuring a long-term vision. The example of Slovenia shows that political stability creates a climate conducive to successful education policies. For example, according to the national stakeholders consulted, one of the important factors that contributed to the success of the 1996 education reform in Slovenia was the fact that no government change took place before and after the reform, thus allowing its smooth planning and implementation. Pointers for policy It is important to restructure the mechanisms of national policy-making to overcome the cyclical nature of reform processes. Government needs to set a general direction, but educational decisions relating to matters such as curriculum content and national testing should be allocated to a non- political body that is set up for a long duration. Prerequisites for the successful functioning of such a body are: experience in managing similar institutions in the country; well-developed educational discourse in the country; sufficient potential for educational research and development has been accumulated; sufficient implementation capacity of the body, and a degree of autonomy. In advancing on the path of reforms and looking at the long term, policy- makers need to confront the short-term adjustment costs: often, the benefits of better structural policies may take time to materialise. This should not be a deterrent to implementation. On the contrary, reformers need to act decisively, as the cost of the status-quo is even greater. → they focus on both equity and quality, rather than prioritising one at the expense of the other. While some countries (e.g., Portugal and Estonia) managed to effectively unite these objectives into a comprehensive set of complementary measures, in other countries one may suffer at the expense of the other. For instance, the increased support for independent schools in Sweden, while to some extent promoting innovation and raising educational standards, is also likely to have resulted in increased school segregation. In this respect, the cases of Estonia and Portugal demonstrate the importance of a long-term holistic vision on inclusion being integrated together with accompanying measures that target vulnerable groups. Contrastingly, the decentralisation and voucher reform in Sweden provides an important lesson in the oversight of equity when focusing on efficiency and innovation. Pointers for policy Equity and quality of education should be prioritised equally. Education systems, and the pathways through them, need to be designed in a way that both enhances equity and increases students’ success, yet a wide range of policies that hinder equity are still common. 18 June 2022 PISA success stories Pointers for policy Policies that eliminate grade repetition, delay streaming and reduce school segregation help to improve equity and enhance quality. The factor of family background affects student learning outcomes not only directly, but also through school choice policies. Therefore, reforms must also consider that family background tend to increase social and academic segregation in early and general education, and policy-makers need to mitigate this. → they build on the consensus of the range of actors (including students, parents, teachers, employers and trade unions) who also take an active part in enacting and mediating policy change. Inclusive policy design can usually be achieved through a series of consultations and discussions. Including relevant stakeholders into policy design not only helps policy makers to better understand the needs of target groups, but is also crucial to making informed decisions. The reform processes in Latvia and Estonia are good illustrations of both stakeholder resistance when left in the background during the policy process (e.g., language reform in Latvia) and active support (assessment reform in Estonia). A multi-stakeholder approach is key to ensuring inclusive policy design and the subsequent buy-in of key actors implementing the reform. Given the variety of stakeholder groups that are crucial for inclusive and quality education, there is a need to understand the possible mechanisms and channels that allow relevant groups to meaningfully participate in various stages of education provision and the planning of education reform. Pointers for policy Effective communication of long-term objectives is of great importance in making reform happen. Making the case for reform also needs to be based on evidence and on achieved public consensus. This is particularly important in education, where professionals, if not engaged, tend to block reforms. Engaging stakeholders can take time, is complex, and can lead to a lack of results if not well organised. For effective implementation, the main issue is how stakeholders can be engaged in ways that can support the educational change. Several factors have been found to be important in engaging stakeholders, such as higher levels of involvement (partnership and co- decision-making), transparency, and communication. → they are monitored and adapted regularly to address the evolving needs of students and schools, and are context-sensitive. Analysis of the selected reforms demonstrates that education systems often lack the capacity to effectively use and interpret monitoring data. Only a few countries have introduced policy initiatives with built-in monitoring and evaluation systems based on hard evidence. The examples of the TEIP and the National Programme to Promote Educational Success (Plano Nacional de Promoção do Sucesso Escolar, PNPSE) in Portugal demonstrate the efforts of the 19 June 2022 PISA success stories government to use piloting before scaling up the interventions nationally; however, even in these cases, no rigorous evaluation based on experimental research is foreseen. Pilot evaluations are largely based on perception studies. These criticisms are sometimes accompanied by a call for more rigorous, scientifically based evidence to lead to so- called ‘evidence-based policy-making’. Partly in response to this debate, some countries are introducing large-scale randomised control trials (RCTs) as a way of evaluating educational interventions. Pointers for policy Strengthening evidence-based education is imperative. This can be achieved through improved monitoring and evaluation practices, ensuring policy designs that allow for experimentation and evaluation, collection and analysis of national longitudinal student assessment data and investment in research, e.g., through creation and supporting a permanent advisory research-based unit which can be called upon to provide evidenced-based advice on on-going issues. Reinforcing the culture and tradition of evidence-based education policymaking in EU Member States is recommended. The best design for educational research to ensure the causality between a reform and its effects is a comparative design. This implies that the more reliable studies to answer a question on an anticipated effect are the (quasi)experimental studies, were a group benefiting from a reform is compared to a group of students or schools not benefiting from a reform. This also implies that reform implementation should be planned in several steps: piloting and assessment of its results, scaling up, monitoring and adjustment. Designing and assessing an education reform according to the best available evidence has been popularized as the Evidence-Based Education (EBE) movement. According to Slavin (2017, 2019), providing educator-friendly reviews, making available a broad range of proven education programs, and providing resources to help schools to implement proven programs are the three most important levers for implementing evidence-based reform. In the United States and in the United Kingdom, for instance, efforts have been made to provide these resources through Evidence for ESSA, What Works Clearinghouse or the Education Endowment Foundation. Investing in longitudinal studies would strengthen the evidence base for education policy-making. Implementing longitudinal studies covering the whole span of education – from kindergarten to upper secondary or even tertiary education comes with high costs many education systems cannot afford. Some countries tried to increase the capacity of PISA to inform educational policies in a more fine-grained or relevant manner at a reasonable cost through introducing a longitudinal component to PISA and oversampling. 20 June 2022 PISA success stories Introduction PPMI and EQUALE are pleased to present the draft final report for the Study on Successful PISA Stories in the EU: How Some Member States Have Been Able to Improve Their Performance Over Time. Mastering basic skills is a prerequisite for thriving in life, finding fulfilling jobs and becoming engaged citizens. Nevertheless, the EU has not achieve the target set out in its Strategic cooperation framework ‘Education and Training 2020’ (ET2020) to reduce the share of 15- year-olds achieving low levels of attainment in reading, maths and science to less than 15% by 2020, as measured by 2018 data from the OECD’s Programme for International Student Assessment (PISA). The post-2020 framework (Strategic framework for European cooperation in education and training towards the European Education Area and beyond [2021-2030]) reinforces the previously set targets, demonstrating a strong social dimension: “Education and training have a vital role to play (…) at a time when it is imperative that its society and economy become more cohesive, inclusive, digital, sustainable, green and resilient, and for citizens to find personal fulfilment and well-being, to be prepared to adapt and perform in a changing labour market and to engage in active and responsible citizenship”.2 The European Education Area envisions a quality education system that allows learners to master transversal skills, promotes learning mobility and cooperation in Europe and beyond, fosters multilingualism and cultural diversity, and introduces learners to a European perspective that is complementary to national and regional perspectives. The European Education Area also highlights the importance of inclusion and gender equality, as well as highlighting the need the need to ensure that policies and investments in education and training also focus on inclusive green and digital transitions (European Commission, 2020, h). Under the current period of the ET framework (2021-2030), COVID-19 and the green recovery in particular have left a mark on the education priorities at EU level and in the Member States. On the one hand, COVID-19 has aggravated socio-economic disparities in education, especially for those experiencing multiple disadvantages. On the other hand, it added impetus to the acute need to foster students’ mastery of basic skills, in addition to digital and green skills, in order to ensure that no one is left behind in education systems and labour markets across Europe in the ‘recovered’ green economy of the future. In particular, the COVID-19 pandemic has highlighted the challenge of creating education systems that are both flexible enough to account for changes in skills needs and ways of learning during a crisis, and at the same time robust enough to provide equal opportunities for all (Blasko, da Costa, & Schnepf, 2021). The average rate of underachievement in the EU – that is, the proportion of pupils who failed to complete basic mathematics, science and reading tasks, according to PISA in 2018 (i.e. they do not attain Level 2 knowledge in the PISA assessment) – stood at 22.5% in reading, 22.9% in mathematics, and 22.3% in science (European Commission, 2019). Furthermore, at EU level, students’ performance has deteriorated gradually over the 2009- 2018 period (European Commission, 2019). However, some EU countries have been able to improve their PISA results over time, putting in place structural education reforms, such as increasing the autonomy of schools, tackling inequalities from children’s early years, and investing in competence-based curricula and assessments. This study aims to identify which factors may be associated with these positive trends in PISA results, and which reforms may have had an impact on student performance. 2 Council Resolution on a strategic framework for European cooperation in education and training towards the European Education Area and beyond (2021-2030) 2021/C 66/01, OJ C 66, 26.2.2021, p. 1–21 21 June 2022 PISA success stories 1.1. How to read this report The report is structured as follows: Chapter 1 (the present chapter) introduces the report. It provides an overview of the study’s aims, objectives and research questions (Section 1.2), as well as the methodology used, including the scope of the study (Section 1.3.1.), and the rationale behind the country success stories selected (Section 1.3.2). It goes on to outline the steps taken in the research (Section 1.3.3.), and zooms in to look at the data used for the analysis (Section 1.3.4.), and the limitations of the study (Section 1.4.5.). Chapter 2 provides an overview of trends in performance across the EU, as measured by PISA (section 2.1.) Seven EU countries are then selected for in-depth analysis, highlighting interesting positive trends in student achievement, and identifying factors that may be associated with these trends (Section 2.2.). Chapter 3 offers a picture of the pathways taken by the seven selected countries towards improvements in PISA performance over time across different subject domains, taking into account not only student achievement, but also the contexts of equity and inclusion. Each country’s success story begins by presenting important educational context in the country as well as salient policy issues. This is followed by a detailed outlook at the country’s PISA trends over time (based on descriptive statistics). This section also aims to identify policy reforms that may possibly have had a positive impact on students’ academic performance or equity in the country. The analysis then focuses on the potential impact of the selected reforms using complex statistical models (where such analysis was possible), contextualised through consultations with relevant stakeholders in the country, to provide insights into what can explain the success of these reforms (i.e. the key success factors). More detailed analysis of each country, as well as a description of methodological steps taken is provided in Annex 1. Chapter 4 provides reflections on the key lessons to be learnt from the reform journeys in the seven countries. Chapter 5 presents the study’s conclusions and their implications for policy and research. The report also includes two Annexes. Annex 1 provides the detailed country reports. Annex 2 includes a detailed explanation of the methodological approach used. 1.2. Aims, objectives and research questions High-quality education benefits both individual learners and society as whole. For individuals, obtaining a better education often results in lower levels of unemployment, higher lifetime earnings, higher job satisfaction (Riddell, 2016), and higher subjective well- being (Witter, Okun, Stock, & Haring, 1984). Given the potential benefits it offers to society, education can increase the human capital of the labour force, which in turn ensures higher labour productivity (Hanushek & Woessmann, 2010), and can contribute to improving social welfare and equity (Behrman & Stacey, 1997). Educational attainment can also be connected to greater civic participation in politics and democratic processes (Glaeser, Ponzetto, & Shleifer, 2007), and to decreasing crime rates (Lochner & Moretti, 2004). Hence, providing quality education has been an important focus for policy makers across the EU and around the world (Bolden & Tymms, 2020). However, it is rarely possible to rigorously determine whether the reforms have produced the intended effect and contributed to increasing the academic performance of students, due to a lack of large- scale experimental evaluations of implemented educational interventions, as well as limited 22 June 2022 PISA success stories evidence of significant improvements in student performance, as demonstrated by data from the large-scale international student assessments (Bolden & Tymms, 2020). The present study attempts to expand existing knowledge about which factors and education reforms could help to improve educational standards in the contexts of specific countries, by testing various statistical approaches to measuring the effects of particular reforms. This analysis is complemented with the results of the non-experimental quantitative research conducted, as well as stakeholder consultations. Specific objectives The central goal of this study is to analyse the factors that may have contributed to improvements in the quality and equity of education in the selected EU countries, all of which have shown improving trends in students’ academic achievement, as measured by PISA. To achieve this objective, the study: ▪ investigates to what extent individual factors (e.g. migrant background, socio- economic background, gender), as well as factors relating to schools (e.g. school climate, instruction time, school type) and systems (e.g. resources, curricula, student tracking, school autonomy, etc.) can explain differences in educational outcomes, as measured in PISA; ▪ analyses how national reforms may have contributed to improvements in the PISA performance of particular countries over time, and explores the success factors behind these reforms; ▪ formulates policy recommendations to tackle underachievement (as measured by PISA) and improve inclusiveness in the education systems of the selected countries and across the EU. 1.2.1. Study research questions Below, we set out main research questions for the study. These are based on the specific objectives outlined above. In addition to these overarching research questions, we also outline country-specific questions based on the particular developments in PISA performance in each of the selected Member States. These questions are introduced and explained in the respective country sections in Chapter 3, and in the full country reports in Annex 1. TABLE 1. GENERAL RESEARCH QUESTIONS Questions What are the trends in PISA performance over time in the EU-27? Which countries have significantly improved their PISA performance over time (scores across three domains and shares of underachievers)? Which countries have narrowed the underachievement gap (as measured by PISA) over time? What is the share of low achievers among different groups of students (disaggregated by gender, migrant background, and socio-economic status) in the EU-27? What are the differences in achievement among different groups of students (disaggregated by gender, migrant background, and socio-economic status) in the EU-27? How do long-term performance trends compare across difference datasets (PISA, TIMSS and PIRLS)? 23 June 2022 PISA success stories Which factors are associated with improved PISA performance and narrowed underachievement gap? What individual factors (i.e. background aspects, including gender, migrant and socio-economic status) affect levels of PISA performance and underachievement rate, as well as changes in these rates over time in the selected countries? What school-level factors (e.g. school climate, teacher activities, etc.) affect levels of PISA performance and underachievement rates, as well as changes in these rates over time in the selected countries? Which aspects of education systems and policies (e.g. early tracking, teacher professional development systems, structural reforms, etc.) explain the differences in PISA performance and underachievement rates, as well as changes in these rates over time in the selected countries? How do the factors at different levels interact and mediate one another? How do the PISA success stories of the selected countries compare? How do countries compare in terms of positive trends in educational outcomes (as measured by PISA) and the factors that are associated with such trends? What factors can explain these differences? Which national reforms may have contributed to improvements in PISA performance over time in the selected countries, and what are the typologies of these reforms? How do the contexts of specific countries and their implementation processes affect the effectiveness of these reforms? What are the success factors behind the selected reforms? Source: authors. 1.3. Overview of methodological approach 1.3.1. Scope of the study The study explores performance trends over the period 2009-2018 (the last four PISA cycles) and examines the education policy factors that may be associated with the positive trends observed. The time span for the information gathered about key education reforms in the selected countries covers a longer period, as the effect of educational reforms on student performance may only be visible a number of years after reforms were implemented. For most countries, qualitative data collection covers a period beginning in 2000 (the so-called ‘first PISA shock’, which triggered various structural reforms across the EU countries). However, in cases where countries implemented influential reforms before the 2000s, such reforms have also been considered. 1.3.2. Selection of PISA success stories While students’ educational success is usually understood in a broader sense, for the purposes of this study a country can be classified a PISA success story if it managed to improve students’ performance in at least one of the three subject domains – reading, mathematics and science – between 2009 and 2018, as well as decreasing its underachievement rate, while improving the equity of education provision (i.e. the socio- economic gap among achievers either decreased or remained stable). Our selection of countries is thus based on the following criteria: 1. Increase in mean PISA performance – if the country experienced an increase in mean PISA performance from 2009 to 2018 in at least one of the domains – reading, mathematics and science. 24 June 2022 PISA success stories 2. Decrease in underachievement rate3 – if the country shows a positive change (or stability) in the underachievement rate in those subjects in which the country has experienced an increase in its mean PISA performance between 2009 and 2018. It is important to ensure that the increase in the mean performance in PISA is linked to an overall improvement in student achievement in the country, rather than to an improvement in the performance of already high-performing students. Countries were only selected if they did not experience an increase in underachievement rate from 2009 to 2018 in tandem with an increase in mean performance score. 3. All of the countries selected demonstrate a narrowing or no increase in the socio-economic status gap – that is, the difference between the PISA scores of socio-economically advantaged and disadvantaged students. It is important that the education system is inclusive and does not leave behind those students from vulnerable backgrounds who are often over-represented among underachievers. The failure of a system to consider the needs and well-being of all students can further perpetuate existing inequalities in society. Consequently, an increase in mean PISA performance without a decrease in the performance gap between students with different socio-economic backgrounds may indicate that the system tends to exclude certain groups of students. As a result, countries were only selected if their mean PISA performance increased between 2009 and 2018, but they did not experience a widening socio-economic gap in PISA performance. While the case study countries were selected on the basis of PISA trends between 2009 and 2018, PISA trends since the 2000s were also analysed to ensure that the observed improvements were not the result of a country having uncharacteristically low PISA scores in 2009 or 2012, for example, but instead show a gradual improvement. Following their initial statistical analysis and desk-based research, the research team identified seven EU Member States – Bulgaria, Estonia, Latvia, Slovenia, Sweden, Poland and Portugal – that have (somewhat) improved their PISA performance over time, as well as demonstrating interesting developments in education policy since 2000. It must be noted, however, that on the basis of PISA data, very few countries demonstrated significant improvements, while others displayed fairly stable trends over time. This constitutes one of the major limitations to the analysis of the possible effects of reforms (see Section 1.3.5 for a discussion on these limitations). Based on the above criteria, a few other countries, such as Ireland and Italy, qualified to be considered a PISA success story. In view of this, additional criteria were employed to narrow down the selection – countries that had previously been less well researched (such as Latvia and Bulgaria) were prioritised, with a view to providing novel evidence on possible factors that might influence PISA improvements. The table below explains the rationale behind the final selection. 3 Underachievement rate is the proportion of pupils who fail to complete basic mathematics, science and reading tasks, according to the OECD’s Programme for International Student Assessment (PISA) (i.e. those who do not attain level 2 knowledge in PISA assessment). 25 June 2022 PISA success stories TABLE 2. EXPLANATION OF COUNTRY SELECTION Country Trends in PISA over time Educational development Bulgaria - Mean performance in all three domains increased Bulgaria still lags behind in terms of the (BG) between 2006 and 2012 (or 2015 for maths), then quality of education provided. Several decreased until 2018. important challenges hinder the development - Underachievement rates in all three domains of the country’s education system; however, decreased between 2006 and 2012 (or 2015 for maths education policies in recent years have and science), then increased until 2018. focused on improving the quality and equity of - Performance gap between socio-economically the education system (with reforms in the advantaged and disadvantaged students decreased fields of ECEC and the assessment system between 2012 and 2015. perceived as potentially influential). - Gender gap in reading performance increased between 2009 and 2012, and decreased between 2012 and 2018. Estonia - Mean performance in mathematics and reading Estonia has a long tradition of external (EE) increased between 2009 and 2018. assessments, decentralised education - Underachievement in mathematics and reading governance, digitalisation and positive decreased between 2009 and 2018. attitudes towards education. Despite lower - Performance gap between socio-economically education spending (compared with other advantaged and disadvantaged students remained PISA top performers), Estonia managed to stable. achieve the highest results among the EU - Gender gap in performance in mathematics Member States, which makes the country an decreased between 2012 and 2015. interesting case study. Latvia - Mean performance in reading increased significantly Latvia has been focusing on ensuring better- (LV) between 2000 and 2003. quality education by improving educational - Mean performance in mathematics increased standards in key disciplines. Policy makers between 2015 and 2018. have also focused on ensuring greater equity - Mean performance in science increased until 2012, in the education system by tackling language- but decreased between 2012 and 2018 based segregation, with particular regard to - Underachievement rates in all domains have the segregation of Russian-speaking remained generally stable. students, as well as supporting vulnerable - Performance gap between socio-economically students. advantaged and disadvantaged students decreased between 2012 and 2018. - Gender gap in reading performance decreased between 2009 and 2015. Poland - Mean performance in all three domains increased Poland is an interesting case study, due to a (PL) between 2009 and 2018. structural reform of the education system that - Underachievement rate in mathematics decreased took place in 1999. This reform introduced the over time. concept of a “middle school” (a separate - Performance gap between socio-economically school for lower-secondary education). This advantaged and disadvantaged students remained reform replaced basic school (up to grade 8) stable. with separate primary and lower-secondary - Gender gap in performance in reading decreased schools. The reform focused on improving between 2009 and 2018. quality of education and ensuring more equal opportunities. However, this reform was reversed in 2016 (despite its observed positive impact), due to negative public opinion about middle schools. Portugal - Mean performance in all three domains increased Portugal is an interesting case, as the country (PT) between 2006 and 2009. has implemented various strands of reforms - Underachievement rates in all domains decreased that may have helped to improve the quality between 2006 and 2009. and equity of education. In the early 2000s, - Performance gap between socio-economically the country focused heavily on a results-based advantaged and disadvantaged students remained approach to education. However, in more stable. recent years, several programmes have been - Gender gap in performance in reading decreased implemented to improve inclusiveness in the between 2012 and 2015, but increased between 2015 education system. and 2018. 26 June 2022 PISA success stories Slovenia - Mean performance in reading increased between The government in Slovenia has focused on (SI) 2012 and 2018. tackling underachievement and improving - Mean performance in science and mathematics the reading competences of its population remained stable. through programmes in the sectors of - Underachievement rate in mathematics and reading education and culture, which makes Slovenia decreased between 2012 and 2018. an interesting case study. The country - Performance gap between socio-economically restructured its education system in 1990s. advantaged and disadvantaged students decreased Moreover, VET education is very popular in between 2009 and 2018. the country, which makes it different from - Gender gap in performance in reading decreased most other EU Member States. between 2009 and 2018. Sweden - Mean performance in all three domains decreased Sweden is an interesting case study due to its (SE) between 2000 and 2012, and increased between 2012 long-term commitment to providing quality and 2018. education that focuses not only on the - Underachievement rates in all three domains academic performance of students, but also increased until 2012, then decreased until 2018. on their well-being and holistic development. - Performance gap between socio-economically However, the country has a unique advantaged and disadvantaged students remained arrangement regarding private (independent) stable. schools, which has shaped its education - Gender gap in reading performance decreased system over the last two decades and which between 2012 and 2018. may have contributed to the quality of education, but has been detrimental to its equity. 1.3.3. Research process This section provides a brief overview of the key components of the study’s methodology, which has been designed to gather and analyse evidence in relation to the main research questions outlined above. This study employed a mixed-methods approach, featuring the elements of both qualitative and quantitative data analysis, as well as data triangulation throughout the implementation of the project. The activities carried out were as follows: ▪ Initial scoping research and selection of PISA success stories (January – March 2021). During the Inception stage, the team conducted initial desk research and descriptive statistical analysis of trends in PISA performance across the EU, which also fed into the selection of countries for in-depth analysis. The PISA trends were contrasted with trends in achievement observed in other cross-national education surveys4. Initial background research was also conducted into the countries of interest. ▪ Mapping and documentation of educational developments and reforms in the selected countries (April – August 2021). This activity included extensive desk research to map the key education policy developments that have taken place in the seven countries since the 2000s, as well as the potential effectiveness of these reforms, as evidenced by existing research (if any). In addition, primary data were collected (via stakeholder interviews) on the key barriers to and enablers of the successful implementation of the selected reforms that may have had an impact on PISA success (see Annex 2 for more details). ▪ Advanced statistical analysis (April – October 2021). To link these educational policies or reforms with any changes in the PISA data, statistical analyses were conducted, firstly with a descriptive purpose. The analyses performed covered various aspects (e.g. student performance, equity, segregation, school climate), to provide the broadest and clearest view of the situation within each of the selected 4 International large-scale assessment surveys for grades 4 and 8 TIMSS (for years 2003, 2007, 2011, 2015 and 2019) and PIRLS (for years 2001, 2006, 2011, 2016) were referred to and analysed for cross-validation of the trends observed in the PISA survey and for comparison with the observed trends. 27 June 2022 PISA success stories countries. These analyses provided a complete picture of the situation in each country, making it possible to assess whether the targeted policies or reforms be discerned from the variations observed, meaning that the intended reforms had really been implemented (for instance, that a reform designed to reduce grade repetition led to an actual reduction in the share of students repeating a grade). This was an intermediary step before trying to estimate whether the reform had any effect or impact on achievement or equity. The next step taken for each country was to review all major reforms since the 1990s, and, where any presuppositions existed, to identify any that could be linked to changes in PISA outcomes through difference-in-differences (DiD) and propensity score matching (PSM) methods. The detailed steps in this analysis are explained in the relevant country reports (see Annex 1). ▪ Final analysis and reporting (September – February 2020). The main objective of this task was to conduct a transversal analysis of the findings collected via various research activities, to develop evidence-based conclusions and policy recommendations. 1.3.4. Data and measurements All the data used for quantitative analyses were extracted from the OECD PISA data sets (PISA cycles 2000, 2003, 2006, 2009, 2012, 2015 and 2018)5. The research team mainly focused on data from 2009 to 2018. However, data since the 2000s was analysed to better understand the long-term trends in each country’s PISA performance. As the data sets for the students and the school questionnaire are separate, they were merged for each cycle, to ensure a full data set that would encompass all student and school-level variables. All statistical analyses were conducted using SAS 9.4. As the PISA samples are not simple random samples, methodological procedures for such samples (including replications and plausible values) were respected. These procedures enabled the research team to calculate the standard error for each parameter, and thus determine whether this parameter is either statistically significant (at the significance level of 0.05, indicating that there is a 5% risk of concluding that a difference exists when there is no actual difference), or is not significant. The full list of the variables used for the statistical analysis is provided in the Annex 2 – Methodological approach section 2.1.3. Variables used for statistical analysis. 1.3.5. Study limitations As with any research study, this report is subject to specific limitations in terms of data and analysis, which should be taken into account when interpreting its findings. These include: ▪ Limitations relating to the definition of a success story in PISA. The focus of this study is to analyse what factors may be associated with improved PISA performance over time; however, a very limited number of countries have demonstrated statistically significant improvements. Furthermore, those improvements that have been documented are very modest (a few percentage points), which also makes it difficult to statistically attribute increases in performance results to the effects of certain variables. ▪ Limitations in the assessment of the effectiveness of education policies and refo

Successful PISA Stories in the EU (2022) PDF

Document Details

Tags

Related

Summary

Full Transcript

Upgrade to continue