What is Published in Impact Journals on School Effectiveness? A Systematic Review of Research Results and Methods

A systematic review of 45 articles on school effectiveness (SE) indexed in SJR and JCR (2014–2018) has been conducted. The results obtained show that the quantitative methodology catalyses the greatest number of researches. The articles are mainly published in Q1-Q2 journals with impact factor. The content of the articles reviewed found that, although exist contextual factors that affect the SE, the variables at the school level also have an impact.


Introduction
The study of school effectiveness is a popular theme in educational science. In the scientific panorama, forums that analyse school effectiveness have been created, such as the International Congress for School Effectiveness and Improvement (ICSEI) and journals that focus on this subject, such as the School Effectiveness and School Improvement (SESI) (Reynolds et al., 2014). School effectiveness is a concept that has evolved throughout history. The Coleman Report (Coleman et al., 1967) opens the door to the study of school effectiveness and large-scale research. It concludes that the socioeconomic characteristics of the student body explain the greater variance in the results. However, the characteristics of schools also affect academic achievement (Azpillaga, Intxausti & Joaristi, 2014;Cohodes, 2016;Fuller & Hollingworth, 2014;Hofman, Hofman & Gray, 2015;Karakolidis, Pitsia & Emvalotis, 2016;Martínez-Abad, Lizasoain, Castro-Morera & Joaristi, 2017;Murillo, 2007;Page, Martín, Orellana & González, 2016;Ranjan, 2014;Schmidt, Burroughs, Zoido & Houang, 2015;Tichnor-Wagner, Harrison & Cohen-Vogel, 2016;Weber, 1971), although the variance explained varies from country to country (Costa & Araújo, 2018;OECD, 2019;Schmidt et al., 2015). For this reason, school effectiveness is currently defined as the impact that the school has on the academic performance and social development of the students (Fuller & Hollingworth, 2014;Murillo, 2007;Reynolds et al., 2014;Scheerens, 2000). On the other hand, effective schools are those that develop students in a comprehensive manner, both in basic instrumental skills and in emotional skills, beyond what would be expected in light of their context, personal characteristics and previous results (Murillo, 2005;Stoll & Fink, 1996).
Education systems must guarantee the effectiveness of their schools, because the benefits go beyond training in basic skills (OECD, 2017a). However, not all schools guarantee quality education (OECD, 2017b). Some countries have translated the concern to optimise their country's education into institutions that seek to improve school efficiency (Childs & Russell, 2017). In the Spanish and Latin-American contexts, large-scale educational performance assessments such as the Programme for International Student Assessment (PISA) have had an impact on educational policies (Kauko, Centeno, Candido, Shiroma & Klutas, 2016;LOMCE, 2013).
Standardised tests have not been exempt from criticism. Limitations of assessments, economic focus or loss of humanism are arguments detrimental to standardised assessments (Deng & Gopinathan, 2016;Méndez, 2014;Parra, 2018). Nevertheless, evaluations of education systems allow for their progress (Ferrão, 2014). The availability of standardised test results and questionnaires on the socio-familiar context allow to establish advanced statistical models (Cordero & Manchón, 2014;Feldhoff, Radisch & Bischof, 2016;Gamazo, Martínez-Abad, Olmos-Migueláñez & Rodríguez-Conde, 2017;Jennings, Deming, Jencks, Lopuch & Schueler, 2015;Reynolds et al., 2014). Value-added statistical models make it possible to find schools with a high residual value (Gamazo, et al., 2017;Iyer & Moore, 2017). The residual gain consists of the difference in score expected for its socioeconomic context and that actually obtained. This parameter makes it possible to locate both high or low-efficiency schools, when exceeding the expected score, and low efficiency schools . The OECD (2017b) recommends helping disadvantaged schools by optimizing their personal and financial resources. Schools can be optimised by knowing the variables at the school level that differentiate effective schools Rutledge, Cohen-Vogel, Osborne-Lampkin & Roberts, 2015). To detect the educational patterns that characterise high and low efficiency schools, some authors start from diagnostic evaluations to identify high or low residue centres Intxausti, Etxeberria & Bartau, 2017;Intxausti, Joaristi & Lizasoain, 2016;Joaristi, Lizasoain & Azpillaga;Lizasoain, Bereziartua & Bartau;. In a first phase, by means of multilevel hierarchical models and taking as a reference the residual gain or the growth of scores, a classification of the centres according to the level of school efficiency achieved over the years is carried out. In other words, a longitudinal perspective (Pedroza-Zúñiga, Cetzal, Surema & Lizasoain, 2018) is taken into account. This diagnostic phase is followed by a second phase in which the centres with the highest and lowest residual gain are studied qualitatively. As a result of this study, the educational guidelines that explain the effectiveness (or ineffectiveness) of the schools are established. This type of methodological design is closer to the reality and complexity of education systems (Feldhoff et al., 2016) because they consider both the personal and contextual characteristics of the students and the schools, as well as the processes that take place in the schools (Scheerens, Luyten & van Ravens, 2011) that could explain their added value .
Therefore, the following research questions are posed: 1) What research methodologies do journals catalyse for school effectiveness? 2) What are the variables that explain school effectiveness? In order to answer the first question, the aim was to find out the methodological trends of articles published on school effectiveness in journals indexed in JCR (WoS) and SJR (Scopus), in the period between 2014 and 2018. Furthermore, in order to answer the second question, the objective of describing the results that explain school effectiveness in these articles was proposed. This objective is addressed as a response to the demand raised in recent research on school effectiveness (Hajisoteriou, Karousiou & Angelides, 2018;Scheerens, 2014;.

Materials and methods
The method used in the research was the systematic review, which allows deepening the study of a subject through gathering, systematizing and relating the results of research conducted (Higgins & Wells, 2011;Manterola, Astudillo, Arias, Claros & Mincir, 2013;Martimbianco et al., 2020;Prendes-Espinosa, García-Tudela & Solano-Fernández, 2020). The methodology was adapted to the present investigation using the following techniques: thematic analysis (Braun & Clarke, 2006), systematic conceptual review (Feldhoff et al., 2016;Hallinger, 2014) and narrative synthesis (Kauffman, 2015). The criteria used for the selection of articles that form part of the systematic review were: • The work should focus on school effectiveness.
• High-impact articles: high-impact articles were defined as those published in journals indexed in the JCR (WoS) and SJR (Scopus) databases. This requirement was to ensure the scientific quality of the articles.
• The period studied was limited to 2014-2018.
• The key search words were School Effectiveness, School Improvement, Educational Multilevel Analysis, PISA and School Value-added.
The systematic review process began with a search of ERIC, WoS and Scopus as the main sources of work collection. High-impact articles were situated in the period 2014-2018. The initial result was 67 selected articles. After a first exploratory analysis, those that did not meet all the established selection criteria were discarded. An inductive thematic analysis of the studies was used with the selected articles (Braun & Clarke, 2006). Two themes that influence school effectiveness were identified: both contextual and school variables. Again, those that did not respond to the selected topics were discarded. The final result of the articles to be analysed was 45 (dx.doi.org/10.17504/protocols.io.bf34jqqw).
After this filter, a systemic conceptual review (Feldhoff et al., 2016;Hallinger, 2014) that allowed us to carry out a descriptive analysis of the research was performed. The variables studied were the approach (qualitative, quantitative or mixed), the impact quartile to which they belong and the data analysis techniques used in this research.
In addition, a narrative synthesis (Kauffman, 2015) of the research results was carried out. These were synthesised and pooled in order to draw new conclusions and expose the literature review. In order to guarantee the quality of the systematic review, the indications on the PRISMA checklist were considered (Moher, Liberati, Tetzlaff & Altman, 2009). Figure 1 shows the percentages over the total number of articles, organised by methodological approach according to impact quartile. The impact quartile was divided into Q1-Q2 and Q3-Q4 for this analysis. The total results in Figure 1 should be interpreted taking into account that the articles indexed in JCR also belong to SJR. These results revealed that quantitative research was more present in impact journals with a prevalence greater than four times. Easy access to the results of large-scale evaluations may have been one of the reasons (Cordero & Manchón, 2014;Feldhoff et al., 2016;Gamazo et al., 2017;Jennings et al., 2015;Reynolds et al., 2014;Scheerens, 2014).

Systematic conceptual review of articles on school effectiveness
On the other hand, in both JCR (25) and SJR (36), the impact factor of the journals was mainly distributed in Q1-Q2, with a prevalence of three and four times respectively, as opposed to Q3-Q4. From this, it could be inferred that the subject of school effectiveness was of interest and impact in educational research in the period studied. Table 1 distinguishes data analysis techniques in the case of quantitative methodologies and data collection in qualitative designs. Quantitative analysis techniques were divided into multilevel design (when considering the nested structure of the data), value-added designs (without considering the nested structure of the data), descriptive designs and hypothesis contrast, structural equations, regression analysis and others. The most commonly used quantitative data analysis technique was a multilevel design. Multilevel designs respect the nested structure of the data. In education, student results are nested in classrooms, schools and countries. Multilevel designs make it possible to evaluate the added value of schools by taking into account the nested structure of the education system (Gamazo et al., 2017). Depending on the research objective, it was used with different nuances. More traditional value-added models underestimate the effect of the centre on school effectiveness (Troncoso, Pampaka & Olsen, 2016). Descriptive-contrast of hypotheses was considered when it was the main tool of the article, not as a complement to other techniques, finding the use of ANOVA (Gaertner, Wurster & Pant, 2014) or the "t" value (Ranjan, 2014). In the regressions, specificities such as quartile regression (Li & Konstantopoulos, 2017) or ordinary squares regression (Hobbs, 2016;Schmidt et al., 2015) could be highlighted as examples. In other quantitative analysis techniques, there were decision trees, quantile value-added case, and the free model according to Page (2017), CLASS-S (an observational instrument), agent-based model and econometric analysis techniques (index of school performance).
In qualitative techniques, meta-analysis, and category analysis were the most used. The specific analysis technique used was not always mentioned. Only a few of the authors cited it: a systematic conceptual review (Feldhoff et al., 2016) and realistic synthesis (Ehren, Eddy-Spicer, Bangpan & Reid, 2017). In other techniques, although the process is very detailed by Rutledge et al. (2015), he does not specify the name of the technique although it could be catalogued as content analysis. On the other hand, mapping was used in combination with the critical incident technique (Day, Gu & Sammons, 2016).

Narrative synthesis of research results on school effectiveness
The results of the narrative synthesis of the 45 papers analysed are presented below. The results were organised into inputs and processes (Scheerens et al., 2011). The inputs describe the results that could explain the effectiveness of the personal and contextual characteristics associated with the students. The processes develop the results that the different authors indicated improved the effectiveness of the schools.
The location of the school was a predictor of educational performance (Cordero & Manchón, 2014;Kelcey & Shen, 2016;Sulis & Porcu 2014) and of the probabilities of entering university (Jennings et al., 2015). This was probably no more than a spurious correlation determined by socioeconomic level (Gamazo et al., 2017;Karakolidis et al., 2016;Özdemir, 2016;Troncoso et al., 2016). For the same reason, the social networks of students and school efficiency were related (Hobbs, 2016;Salgado et al., 2014). Likewise, the greater efficiency of private schools would be explained by the socioeconomic level of the students who attend these schools (Hobbs, 2016;Hofman et al., 2015;Joaristi et al., 2014;Troncoso et al., 2016). Families with a high ESCS will be offered greater learning opportunities to their children (Hobbs, 2016;Schmidt et al., 2015), which would explain that greater book ownership has a positive influence on performance (Evans, Kelley & Sikora, 2014). This could be interpreted as really being the cultural level of the family, rather than the economic level, which would explain a higher percentage of variance with respect to educational performance.
Gender was controversial. There were studies that argued that it does not influence performance (Salgado et al., 2014) and others that do. The female gender negatively affected performance in mathematics in adolescence (Gamazo et al., 2017;Hofman et al., 2015;Karakolidis et al., 2016;Özdemir, 2016;Troncoso et al., 2016) and positively influenced, although with less intensity, reading (Gamazo et al., 2017). Female students benefited more from attending schools with a high socioeconomic level (Van Hek et al., 2018). Again, the socioeconomic context could explain the differences between educational opportunities among both genders.
The repetition condition of the student body, or if the student has changed schools many times, impaired educational performance (Gamazo et al., 2017). Likewise, others personal characteristics of the student, such as self-esteem, learning strategies or perception of drug use, were another factor to consider (Martinez-Abad & Chaparro-Caso, 2017).
Although contextual and personal factors associated with the pupils influenced performance, there are school characteristics that influenced academic outcomes Intxausti et al., 2016Intxausti et al., , 2017Joaristi et al., 2014;Lizasoain et al., 2016). In addition, learning opportunities would explain 37% of the inequalities associated with ESCS (Schmidt et al., 2015) so those schools that are more effective will be able to reduce the inequality gap in the students (Valenzuela et al., 2016).

School factors associated with school effectiveness
The variables associated with the school itself and the processes that take place in it significantly affect the achievements of the learners (Hofman et al., 2015;Schmidt et al., 2015). The school influences the performance of its students from the first years of schooling (Ferrão, 2014). There are centres that have a high added value, regardless of the socioeconomic context in which they are inserted Cohodes, 2014;Ferrão, 2014;Gamazo et al., 2017;Intxausti et al., 2016Intxausti et al., , 2017Jennings et al., 2015;Joaristi et al., 2014;Lizasoain et al., 2016;Martínez et al., 2017;Rutledge et al., 2015). Therefore, educational research should focus its efforts on studying the school variables that affect educational performance (Ferrão, 2014;Joaristi et al., 2014) which cannot always be done in a quantitative way (Fuller & Hollingworth, 2014). The influence of the school could explain 20% of the variance in educational performance (Hofman et al., 2015;Kelcey & Shen, 2016); therefore, the control of these variables would make it possible to optimise the teaching-learning processes and move towards more equitable education. The characteristics are presented in 4 categories: educational approach, educational practice, management team and learning culture.

Educational practice
In terms of educational practice, teachers in effective schools had a greater social and emotional involvement with students and their families. Teachers externalised it as high expectations for students and greater involvement in extracurricular activities (Rutledge et al., 2015). Teachers optimised the use of resources, whether external or internal and systematically help students through extracurricular support or volunteer systems to improve education (Intxausti et al., 2017). There was also greater involvement with student assessments (Ranjan, 2014). These were more frequent and regular in the review of homework and tasks, which was related to better results (Murillo et al., 2014). The highly efficient schools showed a more holistic perspective, attending to the academic and emotional development of the students (Rutledge et al., 2015). In addition, teachers showed more planning and closeness skills (Ranjan, 2014).
The involvement of families in the school seemed to be a defining characteristic of centres with high efficiency Hajisoteriou et al., 2018), with mothers having a great leading role in education (Cordero & Manchón, 2014). In centres with high efficiency, greater sensitivity was observed towards the families of the students, as well as greater connection with them (Rutledge et al., 2015).
There was a discrepancy in the number of students per class, as there was research stating that it did not influence results (Li & Konstantopoulos, 2017). In contrast, Egalite and Kisida (2016) stated that the results worsen as the ratio and educational level increased. However, high-performing schools may be more responsive to challenges because they have better resource planning (Intxausti et al., 2017).

Management team
The school inspections did not appear to identify processes for improving the effectiveness of schools (Gaertner et al., 2014). However, management, in determining different processes, has an influence on school activity (Prasertcharoensuk & Tang, 2017). The management of high-efficiency schools was sensitive to the needs of the students, flexible to change, open to innovation, and exercises democratic leadership (Day et al., 2016;Intxausti et al., 2016). Principals did not reduce school effectiveness to performance on standardised tests. Management used these tests for school improvement (Intxausti et al., 2016). In addition, the management team was involved with the values of the school community, encouraged the desire of the entire educational community to learn and, when they belonged to schools in disadvantaged contexts, carried out greater involvement and saw difficulties as a challenge to be overcome (Day et al., 2016).

Learning culture
The culture of learning in high-efficiency schools had its particularities: there were more opportunities for participation, formal communication structures, collaborative decision-making processes, shared leadership and high expectations among students and teachers (Intxausti et al., 2016;Maroufkhani, Nourani & Boerhannoeddin, 2015;Tichnor-Wagner et al., 2016). In them, beyond the desire to raise scores on standardised assessment tests, there was an intrinsic motivation for improving education in general. The levels of effort and involvement in achieving this purpose were significantly higher in this type of school (Intxausti et al., 2016).
The permanent training of teachers was optimal in high efficiency schools . On the other hand, low efficiency centres justified low results by pointing to external factors and ineffective processes of training and innovation . Permanent training in high efficiency schools was characterised by  systematic implementation, a high degree of involvement located in the school itself (e.g. cascade training) and transfer to practice, which was facilitated by the school and always evaluated. On the other hand, research training of teachers and recurrent reflection on their own practice benefited schools (Mincu, 2015). These skills could make teachers the drivers of change in educational institutions.

Educational approach
The comprehensive education of the human being was the commitment of the teaching staff and was a key element in the improvement of schools with cultural diversity (Hajisoteriou et al., 2018). There were no discriminatory practices in high-efficiency schools. Awareness among students with special educational needs was higher. Any student who needs support in his or her education was considered to have special educational needs. Primitive definitions of this aspect, which focus more on disability, were not contemplated. Furthermore, the teaching staff had deeply rooted values, based on strong religious or social beliefs (Intxausti et al., 2017). It was clear that teachers in highly effective schools had a proactive attitude towards education, with empathy towards other educational agents and a vision of knowledge that was flexible and under constant review (Hajisoteriou et al., 2018;Intxausti et al., 2017).
The composition of the school's pupils also affected school efficiency. The diversity of genders is positive for effectiveness, perhaps because they had better behaviour (Ning, Van Damme, Van Den Noortgate, Yang & Pant, 2015;Van Hek et al., 2018). The classroom climate was one of the factors affecting performance when perceived positively, along with autonomy in the use of technologies or the adaptation of teaching (Gamazo et al., 2017). Classroom discipline would explain 11% of the variance between schools in reading achievement (Ning et al., 2015). More discipline was associated with better outcomes, although these levels varied from country to country. This could be explained, according to Ning et al. (2015), by a positive relationship between socioeconomic level, discipline and results. However, Rutledge et al. (2016) noted that shared values and personal connections affected effectiveness more than educational instruction.

Discussion and Conclusions
The main objective of this research was to understand the methodological trends and describe the results that explain the school effectiveness of the articles published on school effectiveness in the journals indexed in JCR (WoS) and SJR (Scopus), in the period between 2014 and 2018. For this purpose, a systematic review was conducted.
During the period 2014-2018, quantitative studies (69%) captured the bulk of research on school effectiveness. Multilevel designs were the most used in quantitative research. Availability of large-scale assessment results combined with contextual questionnaires (Cordero & Manchón, 2014;Feldhoff, et al., 2016;Gamazo et al., 2017;Jennings et al., 2015;Reynolds et al., 2014) facilitated quantitative studies with advanced statistical techniques. Qualitative and mixed designs were under-represented (16% in both) in JCR and SJR. This could be because high-impact journals would have biases in accepting these paradigms (Fernández-Navas & Postigo-Fuentes, 2020). Some authors suggest that the positivist paradigm is better valued (Fernández-Navas & Postigo-Fuentes, 2020;Fielding, 2020;Maxwell, 2019). However, in the area of school effectiveness, qualitative contributions would make it possible to go deeper into axiological and cultural issues of schools (Rutledge et al., 2015). These cultural and axiological factors could have a greater impact on school effectiveness than other instructional variables (Rutledge et al., 2015), so their knowledge would serve as a basis for the creation of improvement plans (Intxausti et al., 2017;Lizasoain & Angulo, 2014OECD, 2017b. Therefore, the contribution of qualitative and mixed approaches could be beneficial for educational evaluation (Maxwell, 2019;Parra, 2018).
As for the second research objective, the contribution of the 45 articles made it possible to describe and integrate the different results. These were consistent with the previous literature, because although individual and contextual characteristics had an impact on effectiveness, there were also characteristics of the schools that made them more effective Day et al., 2016;Hanushek, & Woessmann, 2017;Intxausti et al., 2016Intxausti et al., , 2017Joaristi et al., 2014;Lizasoain et al., 2014Lizasoain et al., , 2016Rutledge et al., 2015;Tichnor-Wagner et al., 2016). These contributions are summarised in Figure 2. The educational approach was characterised by its inclusiveness, because it optimally addressed diversity; its comprehensiveness, because it considered education as a process of teaching social, emotional and instrumental skills; and its axiology, because values were shared and deeply rooted. The management team made decisions in a democratic way, counting on the opinions of the people involved. It was empathetic, because it sought to understand all solutions; flexible with its decisions and sought educational innovation as a way of improvement. Learning culture was considered participatory, promoting opportunities for participation; with effective communication between members allowing collaborative processes. Educational practice considered the intervention in a holistic and global way, taking into account all the contexts. In particular, it considered the family context, which encouraged its participation in the school. It was also characterised by having high expectations of the students and committing to the educational work of the school.
The characterization of high effectiveness schools can contribute to improving effective schools through the creation of improvement plans based on these results (Intxausti et al., 2017;OECD, 2017bOECD, , 2019Rutledge et al., 2015). Therefore, those approaches that contemplate the selection of high or low value-added schools with their subsequent characterization Day et al., 2016;Intxausti et al., 2016Intxausti et al., , 2017Joaristi et al., 2014;Lizasoain et al., 2014Lizasoain et al., , 2016Rutledge et al., 2015;Tichnor-Wagner et al., 2016), would integrate school effectiveness studies with those of school improvement. In this way, diagnostic assessments could serve both to identify schools with high added value and those that need intervention. However, the results of educational research have little impact on educational policies (Damiani, 2016;Fischman, Topper, Silova, Goebel & Holloway, 2018;Tobin, Nugroho & Lietz, 2016) and on some occasions, the purpose of diagnostic evaluations is not clear (Thessin, 2015), which makes it necessary to put into value the results of educational science for school improvement, given that their consequences have an impact on society (Hanushek & Woessmann, 2020).

Limitation and further perspectives of research
The variables used for the systematic conceptual review were conceptually very open. The complexity and diversity of techniques did not allow a more concrete systematization of the analysis techniques used. It would be convenient to compare in depth the techniques of analysis of current school effectiveness. Certain techniques, such as multilevel hierarchical analysis, were used with particularities present in each one. The information available would make it possible to research the analysis techniques, the samples used and the data collection techniques.
The use of articles indexed in JCR and SJR in a specific period could be a limit to the research and results, as other scientific work was excluded. Therefore, it might be interesting to broaden the perspective in future work. The inclusion of OECD reports could be enriching for future studies.
Finally, the topic was limited to school effectiveness and it was interesting to look more closely at other related topics such as the effectiveness of instruction or the effectiveness of teacher characteristics.

Funding details
This work was funded by the Ministry of Economics and Competitiveness of Spain [EDU2017-84649-P] as an R+D project entitled "Análisis de Centros Escolares Andaluces de muy Alta y muy Baja Eficacia: Pautas para la Mejora Escolar", obtained in a competitive call corresponding to the State Plan of Advancement of Scientific and Technical Research of Excellence, State Subprogram of Generation of Knowledge, of the Secretary of State for Research, Development and Innovation. It has also been funded by the University of Seville's Sixth Research and Transfer Plan (VIPPIT).