CHAPTER 24 Experimental and Quasi-Experimental Approaches in Comparative and International Education Miriam Broeks REAL Centre, University of Cambridge Anthony J. Onwuegbuzie REAL Centre, University of Cambridge Pauline Rose REAL Centre, University of Cambridge Ricardo Sabates REAL Centre, University of Cambridge Introduction In recent years, there has been an increased demand for evidence on the intended short- and long-term effects of educational policies and practices to inform decision-making. In this regard, national governments and international organizations increasingly fund experimental and quasi-experimental research methodologies used to evaluate the impact—or ‘what works’—and to assess the effectiveness of educational policies or programs, with the ultimate aim of improving outcomes (Snilstveit et al., 2016; World Bank, 2020). These methodologies aim to draw conclusions about the cause-and-effect of change interventions. They do so by facilitating comparisons between groups affected and not affected by the educational policies or interventions. The ability of experimental and quasi- experimental designs to make such casual identification is greater than other quantitative research approaches based on cross-sectional or longitudinal designs. The attention given to these approaches is reflected in an increased number of studies published in academic journals that involve the use of experimental approaches and quasi-experimental approaches since 2010 (Connolly et al., 2018; Gopalan et al., 2020). In what follows, we describe experimental and quasi-experimental research approaches, including randomized controlled trials (RCTs). We provide some examples in the comparative and international education (CIE) literature to illustrate benefits and challenges associated with them. Finally, we discuss future applications for experimental and quasi-experimental research designs, highlighting the importance of using a mixed methods approach to CIE research, wherein both quantitative and qualitative approaches are used at every phase of the research process. Methodological Overview Experimental and quasi-experimental designs are used to help make causal inferences. For instance, the learning outcomes of primary school children who are taught using a new teaching method (i.e., intervention or treatment group) could be compared to that of primary school children who are taught using the traditional method (i.e., control/comparison group). Experimental and quasi-experimental designs first seek to establish the comparability between children in the intervention group with those from the control group prior to the intervention. If after giving the intervention to the treatment group the learning outcomes (or other outcomes of interest) differ between children in treatment and control schools, and if there are no plausible rival hypotheses (i.e., credible/logical alternative explanations) that might account for these differences, then the researcher can infer that the intervention has caused the observed changes. Experimental and quasi-experimental designs both involve studies conducted either prospectively (i.e., involving the collection of data on events that occur after the study starts) or retrospectively (i.e., involving the analysis of data or events that have already occurred). The main distinction between experimental and quasi-experimental designs is that experimental designs involve the use of random assignment prior to the intervention, whereas quasi-experimental designs do not. Random assignment means that study participants have the same chance of being assigned to any of the groups of the experiment. The RCT is a specific type of experimental design that involves random assignment of participants either to an experimental group or to a control group. Although an RCT must have a control group, an experimental design does not have to include one—for example, it can describe an experiment involving a comparison of multiple treatment groups without a control group. This random assignment is one of the key conditions for examining the causal link between the intervention and the observed changes in outcomes of interest, such as changes in learning outcomes (see Chapter 7 on Quantitative Methodology). By contrast, a quasi-experimental design is used to estimate the (causal) impact of policies or programs on a target population without using random assignment (Gopalan et al., 2020). For instance, when studying the impact of a school building programs, the fact that building schools does not happen in all places at the same time, produces the sources of variation that are required to estimate differences between those who may now benefit from having a school nearby and those who do not (Chatterjee, 2020). Similarly, when studying reforms that increase the school compulsory leaving age, one cohort is required to stay in education for an extra year whereas previous cohorts can leave education early (Elsayed, 2019). This allows making comparisons in between the educational outcomes of the different cohorts. Nonetheless, because of the inherent lack of randomization in quasi-experiments, the treatment and control groups might differ substantially in terms of composition, that is, socio-demographic, cultural and economic background, before the policy or intervention is implemented. This can make it difficult to demonstrate a causal link between the intervention and observed changes in outcomes. For this reason, a key feature of quasi- experimental designs is the ability to account empirically for variations in such background characteristics, which allows to accurately explain the differences between individuals affected by the policy or intervention and those who were not. When these factors are unaccounted for, the ability to make causal claims with quasi-experimental designs is jeopardized. Methodologically, difference-in-differences, regression discontinuity, or instrumental variables have all been used to identify the (causal) impact of policies or programs. It is important to note that these kinds of quasi-experimental approaches are also known as natural experiments, but these are not experiments in the sense of a random allocation. Instead, they allow separating factors external to the policy or program under study or changes in participation. By doing so, changes can be attributed to the policy or intervention under certain circumstances. Randomization in experimental designs and RCTs helps to reduce the number of plausible rival hypotheses, thereby reducing the number of threats to what is known as internal validity. Internal validity refers to the extent to which changes observed in a study can be confidently attributed to the intervention itself, rather than to other factors. At least 29 threats to internal validity have been identified (Onwuegbuzie, 2003). Table 24.1 provides examples of internal validity threats that are particularly relevant to CIE research. Although not guaranteed, randomization is significantly more likely to reduce internal validity threats because it offers the best chance of the experimental and control groups being similar with respect to socio-demographic, cultural, and economic factors. This, in turn, justifies the use of stronger conclusions regarding causality than would be the case if randomization had not taken place. [Insert Table 24.1 near here.] Table 24.2 presents common experimental and quasi-experimental research designs in CIE research. Most of these designs involve the use of a control or comparison group. When using this design, one must be careful there is no contamination or spillover from the intervention, in other words, that participants in the control group are not indirectly affected by the intervention (e.g., avoiding that participants in the control group interact with participants in the intervention group). In addition, these designs should capture information about the settings where the interventions are taking place, in particular the national and regional policy contexts as well as the ways in which schools operate and school stakeholders interact with pupils and communities. [Insert Table 24.2 near here.] Experimental designs have tended to be viewed by many researchers, including those working in CIE, as being the quantitative method that enables making causal inferences. However, as noted by Krauss (2018), in practice, there are many potential limitations that have been identified in its application. These limitations relate to the following four elements: (a) sampling, (b) intervention implementation, (c) data collection/analysis process, and (d) data interpretation. Regarding sampling, one limitation that generates bias arises from an insufficient sample size to yield statistically reliable results. Known as small sample bias, this unreliability might arise when applying an intervention to a population that is relatively small (e.g., students with disabilities, whereby many remain out of school). Another sampling-related challenge stems from some/most/all participants knowing they will participate in the intervention. This knowledge could lead to participants selected for the intervention group putting more effort merely because they feel special (i.e., Hawthorne effect) or having increased motivation solely because they are involved in a different or novel intervention (i.e., novelty effect); and/or participants selected for the control group feeling demoralized and putting less effort (i.e., resentful demoralization) or feeling threatened and thus deciding to compete with those in the intervention group by expending extra effort in order to demonstrate their competence to a greater extent than would have occurred if they had not been in the control group (i.e., John Henry effect). Limitations associated with intervention implementation that yield bias include situations where the set of preconditions needed for the intervention to work in the RCT and/or another context are not fully met (e.g., students being faced with an unexpected situation during the intervention such as COVID-19, conflict, or political instability). This can mean that the intervention cannot be implemented as intended. The implementation of RCTs also requires inputs from both research and implementation teams so that the conditions for the experiment are met. For instance, researchers might need lists of students in the schools as well as their contact details that are not always available or they are only available upon request and following strict ethical protocols. A limitation associated with the data collection or analysis process from which some form of bias ensues can stem from data not being collected at the start of the program for all relevant and known background factors that might influence the outcome, such as when aptitude scores are not measured. An additional limitation is the fact that not all variables can be fully captured quantitatively. For instance, it might be difficult to capture participants’ thoughts, feelings, and interpretations of their experiences using a survey instrument alone. Limitations associated with the data interpretation process that yield some form of bias include the inability to generalize findings to other contexts beyond the population from which the study sample was selected (i.e., low external validity). When a study has low external validity there is a risk of extrapolation bias. Extrapolation bias occurs when adaptation and context-specific information is not considered for future replications of the intervention (see the case study below for an example of a pedagogical intervention from India that was adapted for implementation in Mexico to allow the extrapolation of findings about its effectiveness). Therefore, for RCTs, there is a trade-off between internal validity (making sure that all conditions are met and therefore working in small and controlled settings) and external validity (the ability to generalize from the evidence obtained from such an experiment). Because of these limitations, experimental and quasi-experimental designs should be accompanied by both replications and follow-ups. Replications, which involve the process of conducting additional research that aims to reproduce or to repeat the methods and findings of a previous study in more contexts, are needed to assess the generalizability of the findings. Follow-ups, which involve the process of extending experimental or quasi- experimental studies, are necessary to capture not just long-term benefits, but also unintended consequences (i.e., undesired effects of the intervention). These can include negative impact on learning outcomes or on other factors affecting participants such as higher stress levels. Replications or follow-ups seldomly happen in the literature due the high costs of conducting experimental designs and, in particular, RCTs. There are important questions around the ethics of some participants receiving an intervention and others not receiving the intervention, especially in cases where the control group condition yields negative outcomes (i.e., unintended consequences). Where there is previous evidence showing the superiority of an intervention in the specific context, an RCT that ignores this information to prioritize having a control group might not comply with the prevailing ethical standards. Another ethical issue that arises from experimental and quasi- experimental design is that of volunteer participation (even in the context of random allocation). Beneficiaries who chose to participate voluntarily could pose a threat to both internal validity and external validity. Therefore, before planning and conducting an experimental or quasi-experimental design, researchers must consider and evaluate all the ethical issues that might arise from the intervention, particularly ethical standards of the context where the experiment is to take place. In some cases, quasi-experimental approaches are preferred over experimental ones to address ethical concerns. This was the case with respect to the multi-dimensional program implemented by CAMFED (Campaign for Female Education) in countries in sub- Saharan Africa, including Tanzania and Zimbabwe. The evaluators of this program chose a quasi-experimental design to assess impact on learning outcomes. CAMFED needed to obtain government permission to be able to collect data for evaluation purposes in districts where CAMFED was not present. There were ethical concerns by government officials and CAMFED regarding randomization if an RCT design were used, given this would mean that neighboring schools would receive differential support. As such, a quasi-experimental design was chosen and control schools were identified in districts with similar characteristics to those CAMFED was supporting (Rose et al., 2022; Sabates et al., 2020). Applications in Comparative and International Education In this section, we discuss applications, advantages, and disadvantages of using experimental and quasi-experimental approaches, drawing on examples from CIE research. To do so, we focus on how these approaches have been used to understand whether there have been changes on outcomes, the processes that led to the change, and system-level changes which may be required to improve outcomes. The section concludes by encouraging the use of experimental and quasi-experimental approaches embedded within broader mixed methods research designs. Outcomes In the context of CIE, experimental and quasi-experimental approaches mainly have been used to evidence whether an intervention impacts educational outcomes. Many studies focus on improving academic outcomes (e.g., literacy or numeracy skills). Some also focus on other outcomes, such as educational progression, completion, or socio-economic outcomes such as earnings and later employment prospects. More recently, there has been a growing focus of studies using these approaches to investigate the impact of interventions on socio-emotional outcomes and critical thinking skills. Experimental and quasi-experimental designs can be used to discern the impact of an intervention on these outcomes between sub-groups (e.g., males vs. females, rural vs. urban, rich vs. poor) or across time. In the case of RCTs, they can also be used to compare how variations of an intervention or different interventions might have differential impact on the same outcomes of interest. Examples of the type of interventions that have been evaluated using experimental and quasi-experimental methods include the impact of pedagogical interventions (e.g., structured pedagogy, teaching at the right learning level of students), to tailored or repeated teacher training, the provision of materials and resources (e.g., textbooks, free school meals, cash transfers, merit-based scholarships, extra schooling time or remedial education), and the use of technology by teachers and students in the classroom (World Bank, 2020). For further examples of interventions evaluated through experimental and quasi-experimental designs, see by Evans and Popova (2015), Snilstveit et al. (2016), and Connolly et al. (2018). To attribute impact, experimental and quasi-experimental approaches seek to account for possible confounding variables (as mentioned earlier, these include socio- demographic, cultural, and economic factors that might explain the differences between those who benefit from the policy or intervention and those who did not). The difficulties around isolating causality in complex and dynamic social settings have meant that these approaches have mostly been used to evaluate the impact of smaller scale interventions that focus on a single context (e.g., an RCT in schools in one country rather than across countries). Nonetheless, the body of evidence from the vast number of experimental and quasi-experimental studies summarized in systematic reviews has been used to draw learning across settings and countries (see Chapter 22 on Systematic Reviews). Process Experimental and quasi-experimental approaches have also been used in CIE research to gain insights on how processes in an intervention drive change. To distill ‘how an intervention works,’ it is necessary to identify the intervention’s key features, such as the intended participants, the required material and human resources (e.g., books, technical equipment, intervention providers, trainers), and the main activities or processes (e.g., order of sessions, amount of the intervention that is received known as dosage, adaptations to the intervention). Variations in the implementation of interventions can affect whether the intervention yields the desired changes in outcomes. Although experimental and quasi-experimental approaches are more often used in relation to outcomes, they can provide some level of information on such processes depending on their design. For example, it is possible to isolate drivers of causality through multi-arm RCTs, where an “arm” is understood as a group of participants receiving different interventions or variations of an intervention. The arms of an RCT can differ by the order in which the activities of an intervention are implemented or include additional or fewer components. Comparisons with the control group and between arms allow assessment of whether one of these versions might lead to impact whereas another does not, or it does so with lesser impact. In this way, it is possible to gain insights on whether there are indispensable components of an intervention, or whether a particular sequence of activities and processes might be preferable. For example, a 3-arm RCT in Indonesia involved the examination of three versions of an intervention that used community social accountability (i.e., that a group of members of the community would hold teachers accountable for their behavior), with the aim to increase teacher performance (Gaduh et al., 2022). One arm included only the social accountability component. The other two combined the social accountability element with either the use of a camera to verify teacher attendance at school by taking a picture at the start and end of the school day, or an evaluation scorecard used by the community to rate teachers. The 3-arm RCT revealed that the evidence provided by the social accountability element with the pictures was more effective in increasing teacher performance and learning outcomes than were the other two arms. System Quasi-experimental approaches have also been used to provide evidence on change at the system-level (Mejía-Rodríguez & Kyriakides, 2022). Educational reforms operate in highly complex social, political, and economic systems. These systems shape the context in which a reform operates and can influence a reform’s success. Given this complexity, RCTs might be more difficult and costly to undertake to evaluate system-level changes. However, quasi-experimental evaluations provide a suitable alternative. As one example, Hoddinott et al. (2019) identified challenges of using experimental methods for evaluating a national quality education reform in Ethiopia given that the government did not consider it appropriate to provide only some schools in the same location with the interventions. A quasi-experimental design was therefore adopted, given that additional support, such as to teacher professional development, was provided to schools identified by government in certain woredas (districts). Quasi-experimental approaches have been used to study various types of reforms through diverse analytical methods. Examples of policies or programs include school building programs (Chatterjee, 2020), changes in school leaving age (Momo et al., 2021), introduction of fee free education (Sabates et al., 2012), among others. In the case of the latter, the introduction of fee-free upper secondary education in Tanzania increased access for some students previously excluded due to financial barriers. By comparing educational or labor market outcomes of cohorts of students who accessed upper secondary prior to the fee-free reform with those who accessed upper secondary education after the reform, the researchers could establish the potential impact of such reform. In terms of analytical methods, as noted in the ‘Methodological Overview’ section, examples include regression discontinuity designs, instrumental variables, and difference-in-differences methodologies (see also Chapter 36 on Regression). Future Possibilities of Experimental and Quasi-Experimental Approaches in CIE To draw learning from one study to another context, it is particularly important to provide rich detail on the intervention processes and activities. However, this is a limitation of many published experimental and quasi-experimental studies (Snilstveit et al., 2016). Although most researchers discuss aspects of the methods used (e.g., randomization, sampling, and contamination issues) in detail, many fall short in providing a comprehensive description of the processes that led to the observed changes in outcomes, a clear description of the interventions themselves, or any challenges to the implementation of the interventions (Evans & Popova, 2015; Snilstveit et al., 2016). Understanding the processes of an intervention can help understand why impact might or might not be sustained after a follow-up, when scaling-up the intervention or when implementing it with a different sample (Deaton & Cartwright, 2018; Evans & Popova, 2015; Gaduh et al., 2022). In addition, experimental and quasi-experimental approaches would benefit from being complemented with insights on students’, teachers’, or parents’ experiences, preferences, or enablers and barriers of the intervention. Likewise, this information can help explain why impact might or might not be sustained over time or with a different sample. To address these issues, mixed methods research approaches to experimental and quasi-experimental, such as Mixed Methods Randomized Controlled Trials (MMRCT; Edmunds et al., 2022), are an important way forward in the future use of these approaches (Chapter 8 on Mixed Methods Research). For example, an RCT in Kenya included qualitative observations to understand how well the intervention on mother-tongue instruction was implemented and what usual practice looked like in the control schools for the duration of the study (Piper et al., 2016, 2018). Findings revealed that, while the use of mother tongue increased in the two intervention arms, a similar increase was observed in control classrooms. Qualitative observation data helped identify that there was resistance to using the intervention in some schools. This helped explain why the intervention did not yield the expected results. Sample CIE Study: Using Experimental or Quasi-Experimental Approaches Across Contexts This section discusses the adoption and evaluation in Mexico of the Combined Activities for Maximized Learning (CAMaL) program, originally developed by Pratham in India with the aim of ‘teaching at the right level’ (Pratham, 2016). The CAMaL program was evaluated using an RCT to successfully raise foundational literacy and numeracy in India (Banerjee et al., 2016; Banerji & Chavan, 2016). Aspects of this teaching methodology have been adopted and evaluated in African and South Asian contexts, where it has been found also to improve learning outcomes (Angrist et al., 2020). In consultation with Pratham, the program was subsequently adapted, implemented, and evaluated in Mexico by researchers from CIESAS1 and the Universidad Veracruzana2. A research team in Mexico, led by Hevia3 and Vergara-Lope4, adopted it for the context of Veracruz, Mexico. This resulted in the Reactivando-MIA program.5 Consultations helped discern areas that could be improved given what was known to work well from prior implementation and evaluation by Pratham. The choice of evaluation design used by the Mexican team was driven by contextual considerations. Although many of the CAMaL studies were RCTs, a quasi-experimental design was deemed more appropriate in this case. This was, in part, due to financial considerations, but also importantly participants for the study—both parents and school directors and teachers—were not keen on an RCT design. For example, the Mexican team identified resistance by school directors to the idea of being on a waiting list as part of the control arm. As a result, an experimental group only pre-/post-test design (i.e., without control group) was used (see also Table 24.2) (Hevia & Vergara-Lope, 2022). Although this design limits the ability to make causal claims on the impact of Reactivando-MIA, this was the most viable route to start building an evidence-base of the adapted CAMaL methodology in this context. This highlights an important issue that not all educational programs or contexts are appropriate for the use of experimental methods, and the utilization of quasi-experimental methods should not be seen as a second-best option. Conclusion Experimental and quasi-experimental approaches are useful for identifying the impact of educational interventions. In recent years, international organizations have adopted these approaches to identify ‘what works.’ Further, they can be used to gain insights on how intervention processes might lead to differing impact in the case of multi- arm RCTs, and quasi-experimental designs are useful for gauging the impact of complex interventions and policy reforms that enact system-level changes. The example from India 1 https://sureste.ciesas.edu.mx/ 2 https://www.uv.mx/ 3 https://golfo.ciesas.edu.mx/hevia-de-la-jara-felipe/ 4 https://www.uv.mx/personal/svergaralope/datos-generales/ 5 https://www.medicionmia.org.mx/ and Mexico highlighted the potential that both experimental and quasi-experimental designs have to increase our knowledge of what works across contexts. Experimental and quasi-experimental approaches have strengths and weaknesses that have been outlined in this chapter. Although they can provide some information on intervention processes and system changes, the depth they can generate is often limited. Therefore, experimental and quasi-experimental studies would benefit from being complemented with mixed methods research approaches to understand more deeply the implementation, intervention processes, and potential unintended consequences of an intervention. Such mixed methods research approaches are likely to lead to improvements on the relevance of experimental and quasi-experimental findings for understanding what works. Foundational Readings Hesse-Biber, S. (2012). Weaving a multimethodology and mixed methods praxis into randomized control trials to enhance credibility. Qualitative Inquiry, 18(10), 876– 889. Spillane, J. P., Pareja, A. S., Dorner, L., Barnes, C., May, H., Huff, J., & Camburn, E. (2010). Mixing methods in randomized controlled trials (RCTs): Validation, contextualization, triangulation, and control. Educational Assessment, Evaluation and Accountability, 22, 5–28. Whatley, M. (2022). Introduction to experimental and quasi-experimental design. In M. Whatley, Introduction to quantitative analysis for international educators (pp. 141– 166). Springer Texts in Education. Applications in CIE Burde, D. (2012). Assessing impact and bridging methodological divides: Randomized trials in countries affected by conflict. Comparative Education Review, 56(3), 448-473. Pridmore, P., & Jere, C. (2011). Disrupting patterns of educational inequality and disadvantage in Malawi. Compare: A Journal of Comparative and International Education, 41(4), 513-531. Yaya, S., Okonofua, F., Ntoimo, L., Kadio, B., Deuboue, R., Imongan, W., & Balami, W. (2018). Increasing women’s access to skilled pregnancy care to reduce maternal and perinatal mortality in rural Edo State, Nigeria: A randomized controlled trial. Global Health Research and Policy, 3(12), 1-10. Zha, Q., & Tu, D. (2016). Doing mixed methods research in comparative education: Some reflections on the fit and a survey of the literature. International Perspectives on Education and Society, 28, 165-191. Additional References Angrist, N., Evans, D. K., Filmer, D., Glennerster, R., Rogers, F. H., & Sabarwal, S. (2020). How to improve education outcomes most efficiently? A comparison of 150 interventions using the new Learning-Adjusted Years of Schooling metric. https://openknowledge.worldbank.org/server/api/core/bitstreams/031d0280-a1dd- 551e-8646-e38da245f529/content Banerjee, A., Banerji, R., Berry, J., Duflo, E., Kannan, H., Mukerji, S., Shotland, M., & Walton, M. (2016). Mainstreaming an effective intervention: Evidence from randomized evaluations of “Teaching at the Right Level” in India. National Bureau of Economic Research Working Paper Series. https://www.nber.org/system/files/working_papers/w22746/w22746.pdf Banerji, R., & Chavan, M. (2016). Improving literacy and math instruction at scale in India’s primary schools: The case of Pratham’s Read India program. Journal of Educational Change, 17(4), 453–475. https://doi.org/10.1007/s10833-016-9285-5 Chatterjee, S. (2020). From better schools to better nourishment: Evidence from a school- building program in India. IZA Journal of Labor Policy, 10(1), 149-160. https://doi.org/doi:10.2478/izajolp-2020-0002 Connolly, P., Keenan, C., & Urbanska, K. (2018). The trials of evidence-based practice in education: A systematic review of randomised controlled trials in education research 1980–2016. Educational Research, 60(3), 276-291. https://doi.org/10.1080/00131881.2018.1493353 Deaton, A., & Cartwright, N. (2018). Understanding and misunderstanding randomized controlled trials. Social Science & Medicine, 210, 2–21. https://doi.org/10.1016/j.socscimed.2017.12.005 Edmunds, J. A., Gicheva, D., Thrift, B., & Hull, M. (2022). Using mixed methods to explore variations in impact within RCTs: The case of project COMPASS. Journal of Mixed Methods Research, 16(4), 478-499. https://doi.org/10.1177/15586898211033144 Elsayed, M. A. A. (2019). Keeping kids in school: The long-term effects of extending compulsory education. Education Finance and Policy, 14(2), 242–271. https://doi.org/10.1162/edfp_a_00254 Evans, D. K., & Popova, A. (2015). What really works to improve learning in developing countries? An analysis of divergent findings in systematic reviews. Policy Research Working Paper No. 7203. World Bank Group. https://openknowledge.worldbank.org/server/api/core/bitstreams/8a259d77-07f5- 58a9-b977-091408db252a/content Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (2022). Does performance pay enhance social accountability? Evidence from remote schools in Indonesia. National Bureau of Economic Research. Gopalan, M., Rosinger, K., & Ahn, J. B. (2020). Use of quasi-experimental research designs in education research: Growth, promise, and challenges. Review of Research in Education, 44(1), 218–243. https://doi.org/10.3102/0091732X20903302 Hevia, F. J., & Vergara-Lope, S. (2022). Efectos de cursos remediales de lectura y matemáticas en logro y desigualdades educativos durante la Covid-19 en México. IE Revista de Investigación Educativa de La REDIECH, 13, e1536–e1536. https://doi.org/10.33010/ie_rie_rediech.v13i0.1536 Hoddinott, J., Iyer, P., Sabates, R., & Woldehanna, T. (2019). Evaluating large-scale education reforms in Ethiopia. RISE Working Paper Series, 19/034. https://doi.org/10.35489/BSG-RISE-WP_2019/034. https://riseprogramme.org/sites/default/files/publications/RISE_WP- 034_ETH_Large_Scale_Reforms.pdf https://openknowledge.worldbank.org/server/api/core/bitstreams/031d0280-a1dd-551e-8646-e38da245f529/content https://openknowledge.worldbank.org/server/api/core/bitstreams/031d0280-a1dd-551e-8646-e38da245f529/content https://www.nber.org/system/files/working_papers/w22746/w22746.pdf https://doi.org/10.1007/s10833-016-9285-5 https://doi.org/doi:10.2478/izajolp-2020-0002 https://doi.org/10.1080/00131881.2018.1493353 https://doi.org/10.1016/j.socscimed.2017.12.005 https://doi.org/10.1177/15586898211033144 https://doi.org/10.1162/edfp_a_00254 https://openknowledge.worldbank.org/server/api/core/bitstreams/8a259d77-07f5-58a9-b977-091408db252a/content https://openknowledge.worldbank.org/server/api/core/bitstreams/8a259d77-07f5-58a9-b977-091408db252a/content https://doi.org/10.3102/0091732X20903302 https://doi.org/10.33010/ie_rie_rediech.v13i0.1536 https://doi.org/10.35489/BSG-RISE-WP_2019/034 https://riseprogramme.org/sites/default/files/publications/RISE_WP-034_ETH_Large_Scale_Reforms.pdf https://riseprogramme.org/sites/default/files/publications/RISE_WP-034_ETH_Large_Scale_Reforms.pdf Krauss, A. (2018). Why all randomised controlled trials produce biased results. Annals of Medicine, 50(4), 312-322. https://doi.org/10.1080/07853890.2018.1453233 Mejía-Rodríguez, A. M., & Kyriakides, L. (2022). What matters for student learning outcomes? A systematic review of studies exploring system-level factors of educational effectiveness. Review of Education, 10(3), e3374. https://doi.org/10.1002/rev3.3374 Momo, M. S. M., Cabus, S. J., & Groot, W. (2021). Evidence on the marginal impact of a compulsory secondary education reform in Senegal on years of education and changes in high school decisions. International Journal of Educational Research Open, 2, 100058. https://doi.org/10.1016/j.ijedro.2021.100058 Onwuegbuzie, A. J. (2003). Expanding the framework of internal and external validity in quantitative research. Research in the Schools, 10(1), 71-90. Piper, B., Zuilkowski, S. S., Kwayumba, D., & Oyanga, A. (2018). Examining the secondary effects of mother-tongue literacy instruction in Kenya: Impacts on student learning in English, Kiswahili, and mathematics. International Journal of Educational Development, 59, 110–127. https://doi.org/10.1016/j.ijedudev.2017.10.002 Piper, B., Zuilkowski, S. S., & Ong’ele, S. (2016). Implementing mother tongue instruction in the real world: Results from a medium-scale randomized controlled trial in Kenya. Comparative Education Review, 60(4), 776–807. https://doi.org/10.1086/688493 Pratham. (2016). Concept note- combined activities for maximized learning. http://img.asercentre.org/docs/Capacity%20building/Kamaal%20ke%20Shikshak/ca malconceptnote.pdf Rose, P., Sabates, R., Delprato, M., & Alcott, B. (2022). Targeted and multidimensional approaches to overcome inequalities in secondary education for adolescent girls: The impact of the Campaign for Female Education (CAMFED) Program in Tanzania and Zimbabwe. Comparative Education Review, 66(4), 733-759. https://doi.org/10.1086/721849 Sabates, R., Westbrook, J., & Hernandez-Fernandez, J. (2012). The 1977 Universal Primary Education Reform in Tanzania: A historical base for quantitative enquiry. International Journal of Research and Method in Education, 55(1), 55-70. https://doi.org/10.1080/1743727X.2011.609551 Sabates, R., Rose, P., Delprato, M., & Alcott, B. (2020). Assessing cost-effectiveness with equity for a programme for marginalised girls in Tanzania. Journal of Development Effectiveness, 13(1), 28-46. https://doi.org/10.1080/19439342.2020.1844782 Snilstveit, B., Stevenson, J., Phillips, D., NatCen, M. V., Gallagher, E., Schmidt, T., & Eyers, J. (2016). Interventions for improving learning outcomes and access to education in low-and middle-income countries: A systematic review. International Initiative for Impact Evaluation, 2015. World Bank. (2020). Cost-effective approaches to improve global learning: What does recent evidence tell us are “Smart Buys” for improving learning in low and middle income countries? https://documents.worldbank.org/en/publication/documents- reports/documentdetail/719211603835247448/cost-effective-approaches-to- improve-global-learning-what-does-recent-evidence-tell-us-are-smart-buys-for- improving-learning-in-low-and-middle-income-countries https://doi.org/10.1080/07853890.2018.1453233 https://doi.org/10.1002/rev3.3374 https://doi.org/10.1016/j.ijedro.2021.100058 https://doi.org/10.1016/j.ijedudev.2017.10.002 https://doi.org/10.1086/688493 http://img.asercentre.org/docs/Capacity%20building/Kamaal%20ke%20Shikshak/camalconceptnote.pdf http://img.asercentre.org/docs/Capacity%20building/Kamaal%20ke%20Shikshak/camalconceptnote.pdf https://doi.org/10.1086/721849 https://doi.org/10.1080/1743727X.2011.609551 https://doi.org/10.1080/1743727X.2011.609551 https://doi.org/10.1080/19439342.2020.1844782 https://documents.worldbank.org/en/publication/documents-reports/documentdetail/719211603835247448/cost-effective-approaches-to-improve-global-learning-what-does-recent-evidence-tell-us-are-smart-buys-for-improving-learning-in-low-and-middle-income-countries https://documents.worldbank.org/en/publication/documents-reports/documentdetail/719211603835247448/cost-effective-approaches-to-improve-global-learning-what-does-recent-evidence-tell-us-are-smart-buys-for-improving-learning-in-low-and-middle-income-countries https://documents.worldbank.org/en/publication/documents-reports/documentdetail/719211603835247448/cost-effective-approaches-to-improve-global-learning-what-does-recent-evidence-tell-us-are-smart-buys-for-improving-learning-in-low-and-middle-income-countries https://documents.worldbank.org/en/publication/documents-reports/documentdetail/719211603835247448/cost-effective-approaches-to-improve-global-learning-what-does-recent-evidence-tell-us-are-smart-buys-for-improving-learning-in-low-and-middle-income-countries