Delphi Study on Standardized Systems to Monitor Student Learning Outcomes in Flanders: Mechanisms for Building Trust and/or Control?

[1] Adler, M., & Ziglio, E. (1996). Gazing into the oracle: The Delphi method and its application to social policy and public health. London: Jessica Kingsley Publishers.

[2] Allal, L. (2013). Teachers' professional judgement in assessment: A cognitive act and a socially situated practice. Assessment in Education: Principles, Policy & Practice, 20(1), 20–34.

[3] Amrein-Beardsley, A., Berliner, D. C., & Rideau, S. (2010). Cheating in the first, second, and third degree: Educators' responses to high-stakes testing. Educational Policy Analysis Archives, 18(14), 1–36. | DOI 10.14507/epaa.v18n14.2010

[4] Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis. Educational Researcher, 36(5), 258–267. | DOI 10.3102/0013189X07306523

[5] Baird, J., Ahmed, A., Hopfenbeck, T., Brown, C., & Elliott, V. (2013). Research evidence relating to proposals for reform of the GCSE. Oxford, UK: Oxford University Centre for Educational Assessment.

[6] Beaver, J. K., & Weinbaum, E. H. (2015). State test data and school improvement efforts. Educational Policy, 29(3), 478–503. | DOI 10.1177/0895904813510774

[7] Bishop, J. (1998). The effect of curriculum-based external exit exams on student achievement. Journal of Economic Education, 29(2), 172–182. | DOI 10.1080/00220489809597951

[8] Black, P., Harrison, C., Hodgen, J., Marshall, B., & Serret, N. (2011). Can teachers' summative assessments produce dependable results and also enhance classroom learning? Assessment in Education: Principles, Policy & Practice, 18(4), 451–469. | DOI 10.1080/0969594X.2011.557020

[9] Brennan, R., Kim, J., Wenz-Gross, M., & Siperstein, G. (2001). The relative equitability of high-stakes testing versus teacher-assigned grades: An analysis of the Massachusetts Comprehensive Assessment System (MCAS). Harvard Educational Review, 71(2), 173–217. | DOI 10.17763/haer.71.2.v51n6503372t4578

[10] Burgess, S., Propper, C., & Wilson, D. (2002). Does performance monitoring work? A review of evidence from the UK public sector, excluding health care CMPO Working Paper Series.

[11] Carnoy, M., & Loeb, S. (2002). Does external accountability affect student outcomes? A cross-state analysis. Educational Evaluation and Policy Analysis, 24(4), 305–331. | DOI 10.3102/01623737024004305

[12] Chong, H., Adnan, H., & Zin, R. M. (2012). A feasible means of methodological advance from Delphi methods: A case study. International Journal of Academic Research, 4(2), 247–253.

[13] Cizek, G. J. (2005). High-stakes testing: Contexts, characteristics, critiques, and consequences. In R. P. Phelps (Ed.), Defending standardised testing (pp. 23-54). London: Lawrence Erlbaum Associates Publishers.

[14] Cobb, F., & Russell, N. M. (2014). Meritocracy or complexity: Problematizing racial disparities in mathematics assessment within the context of curricular structures, practices, and discourse. Journal of Education Policy, 30(5), 631–649. | DOI 10.1080/02680939.2014.983551

[15] Collins, S., Reiss, M., & Stobart, G. (2010). What happens when high-stakes testing stops? Teachers' perceptions of the impact of compulsory national testing in science of 11-year olds in England and its abolition in Wales. Assessment in Education: Principles, Policy & Practice, 17(3), 273–286. | DOI 10.1080/0969594X.2010.496205

[16] Dalkey, N. C., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458–467. | DOI 10.1287/mnsc.9.3.458

[17] Day, J., & Bobeva, M. (2005). A generic toolkit for the successful management of Delphi studies. Electronic Journal of Business Research Methods, 3(2), 103–116.

[18] De Lange, M., & Dronkers, J. (2007). Hoe gelijkwaardig blijft het eindexamen tussen scholen? Discrepanties tussen de cijfers voor het schoolonderzoek en het centraal examen in het voortgezet onderwijs tussen 1998 en 2005. Nijmegen: Netherlands.

[19] Elstad, E. (2009). Schools which are named, shamed and blamed by the media: School accountability in Norway. Educational Assessment Evaluation and Accountability, 21(2), 173–189. | DOI 10.1007/s11092-009-9076-0

[20] Gipps, C., & Stobart, G. (2009). Fairness in assessment. In C. Wyatt-Smith & J. J. Cumming (Eds.), Educational assessment in the 21st century: Connecting theory and practice (pp. 105–118). London: Springer.

[21] Goodman, D., & Hambleton, R. K. (2005). Some misconceptions about large-scale educational assessments. In R. P. Phelps (Ed.), Defending standardized testing (pp. 91–110). Mahaj, NJ/London: Lawrence Erlbaum Associates Publishers.

[22] Haertel, E. H. (1999). Validity arguments for high-stakes testing: In search of the evidence. Educational Measurement: Issues and Practice, 18(4), 5–9. | DOI 10.1111/j.1745-3992.1999.tb00276.x

[23] Haney, W. (2000). The myth of the Texas miracle in education. Education Analysis Policy Archives, 8(41), 1–20. | DOI 10.14507/epaa.v8n41.2000

[24] Horn, C. (2005). Standardised assessments and the flow of students into the college admission pool. Educational Policy, 19(2), 331–348. | DOI 10.1177/0895904804274057

[25] Hoxby, C. M. (2003). School choice and school competition: Evidence from the United States. Swedish Economic Policy Review, 10(1), 11–67.

[26] Hsu, C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1–8.

[27] Janssens, F. J. G., Rekers-Mombarg, L., & Lacor, E. (2014). Leerwinst en toegevoegde waarde in het primair onderwijs. Den Haag: Ministerie van Onderwijs, Cultuur en Wetenschap; Inspectie van het Onderwijs; Rijksniversiteit Groningen; CED Groep; Universiteit Twente; CITO.

[28] Jürges, H., Büchel, F., & Schneider, K. (2005). The effect of central exit examinations on student achievement: Quasi-experimental evidence from TIMSS Germany. Journal of the European Economic Association, 3(5), 1134–1155. | DOI 10.1162/1542476054729400

[29] Jürges, H., Richter, W. F., & Schneider, K. (2005). Teacher quality and incentives: Theoretical and empirical effects of standards on teacher quality. FinanzArchiv / Public Finance Analysis, 61(3), 298–326. | DOI 10.1628/001522105774978985

[30] Jürges, H., & Schneider, K. (2010). Central exit examinations increase performance... but take the fun out of mathematics. Journal of Population Economics, 23(2), 497–517. | DOI 10.1007/s00148-008-0234-3

[31] Jürges, H., Schneider, K., Senkbeil, M., & Carstensen, C. H. (2009). Assessment drives learning: The effect of central exit exams on curricular knowledge and mathematical literacy. Economics of Education Review, 31(1), 56–65. | DOI 10.1016/j.econedurev.2011.08.007

[32] Karsten, S., Visscher, A., & De Jong, T. (2001). Another side to the coin: The unintended effects of the publication of school performance data in England and France. Comparative Education, 37(2), 231–242. | DOI 10.1080/03050060120043439

[33] Keeves, J. P., Hungi, N., & Afrassa, T. (2005). Measuring value added effects across schools: Should schools be compared in performance? Studies in Educational Evaluation, 31(2-3), 247–266.

[34] Klein, E. D., & Van Ackeren, I. (2012). Challenges and problems for research in the field of statewide exams. A stock taking of differing procedures and standardization levels. Studies in Educational Evaluation, 37(4), 180–188. | DOI 10.1016/j.stueduc.2012.01.002

[35] Klein, S. P., Hamilton, L. S., McCaffrey, D. F., & Stecher, B. M. (2000). What do test scores in Texas tell us? Educational Policy Analysis Archives, 8(49), 1–22.

[36] Klenowski, V. (2014). Towards fairer assessment. Australian Education Research, 41(4), 445–470. | DOI 10.1007/s13384-013-0132-x

[37] Linstone, H. A., & Turoff, M. (1975). The Delphi method. Techniques and applications. London, Amsterdam, Ontario, Sydney, Tokyo: Addison-Wesley Publishing Company.

[38] Loeb, S. (2013). How can value-added measures be used for teacher improvement? In Carnegie Knowledge Network (Ed.), What we know series: Value-added methods and applications. Stanford, CA: Carnegie Knowledge Network.

[39] Marlow, R., Norwich, B., Ukoumunne, O. C., Hansford, L., Sharkey, S., & Ford, T. (2014). A comparison of teacher assessment (APP) with standardised tests in primary literacy and numeracy. Assessment in Education: Principles, Policy & Practice, 21(4), 412–426. | DOI 10.1080/0969594X.2014.936358

[40] Mietzner, D., & Reger, G. (2005). Advantages and disadvantages of scenario approaches for strategic foresight. International Journal of Technology Intelligence and Planning, 1(2), 220–239. | DOI 10.1504/IJTIP.2005.006516

[41] Neumann, M., Trautwein, U., & Nagy, G. (2011). Do central examinations lead to greater grading comparability? A study of frame-of-reference effects on the university entrance qualification in Germany. Studies in Educational Evaluation, 37(4), 206–217. | DOI 10.1016/j.stueduc.2012.02.002

[42] OECD. (2013). Synergies for better learning. An international perspective on evaluation and assessment. Paris: OECD Publishing.

[43] Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15–29. | DOI 10.1016/j.im.2003.11.002

[44] Onderwijsinspectie. (2013). Onderwijsspiegel 2013 [Education mirror 2013]. Brussel: Onderwijsinspectie / Vlaams Ministerie van Onderwijs en Vorming.

[45] Perryman, J., Ball, S., Maguire, M., & Braun, A. (2011). Life in the pressure cooker – school league tables and English and mathematics teachers' responses to accountability in a results-driven era. British Journal of Educational Studies, 59(2), 179–195. | DOI 10.1080/00071005.2011.578568

[46] Phelps, R. P. (Ed.) (2005). Defending standardized testing. London: Lawrence Erlbaum Associates Publishers.

[47] Popham, W. J. (2005). Can growth ever be beside the point? Educational Leadership, 63(3), 83–84.

[48] Ramsteck, C., Muslic, B., Graf, T., Maier, U., & Kuper, H. (2015). Data-based school improvement: The role of principals and school supervisory authorities within the context of low-stakes mandatory proficiency testing in four German states. International Journal of Educational Management, 29(6), 766–789.

[49] Ritchie, J., & Spencer, L. (1993). Qualitative data analysis for applied policy research. In A. Bryman & R. Burgess (Eds.), Analysing qualitative data (pp. 173–194). London: Routledge.

[50] Sahlberg, P. (2011). Finnish lessons: What can the world learn from educational change in Finland? New York: Teachers College Press.

[51] Saunders, L. (1999). A brief history of educational 'value added': How did we get to where we are? School Effectiveness and School Improvement, 10(2), 233–256.

[52] Saunders, L. (2000). Understanding schools' use of 'value added' data: The psychology and sociology of numbers. Research Papers in Education, 15(3), 241–258. | DOI 10.1080/02671520050128740

[53] Schildkamp, K., Rekers-Mombarg, L., & Harms, T. J. (2012). Student group differences in examination results and utilization for policy and school development. School Effectiveness and School Improvement, 23(2), 229–255.

[54] Schwartz, P. (1991). The art of the long view. London: Century Business.

[55] Segool, N. K., Carlson, J. S., Goforth, A. N., von der Embse, N., & Barterian, J. A. (2013). Heightened test anxiety among young children: Elementary school students' anxious responses to high-stakes testing. Psychology in the Schools, 50(5), 489–499. | DOI 10.1002/pits.21689

[56] Shepard, L. A. (1992). Will standardised test improve student learning? Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing.

[57] Shewbridge, C., Hulshof, M., Nusche, D., & Stoll, L. (2011). School evaluation in the Flemish community of Belgium. In OECD (Ed.), OECD reviews of evaluation and assessment in education. Paris: OECD Publishing.

[58] Sireci, S. G. (2005). The most frequently unasked questions about testing. In R. P. Phelps (Ed.), Defending standardized testing (pp. 111–121). Mahaj, NJ/London: Lawrence Erlbaum Associates Publishers.

[59] Smagorinsky, P., Lakly, A., & Johnson, T. S. (2002). Acquiescence, accommodation, and resistance in learning to teach within a prescribed curriculum. English Education, 34(3), 187–213.

[60] Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: SAGE.

[61] Tymms, P. (1997). Responses of headteachers to value-added and the impact of feedback. London: School Curriculum and Assessment Authority.

[62] Van Ackeren, I., Block, R., Klein, E. D., & Kühn, S. M. (2012). The impact of statewide exit exams: A descriptive case study of three German states with differing low stakes exam regimes. Education Policy Analysis Archives, 20(8), 1–32.

[63] Vanhoof, J., Van Petegem, P., Verhoeven, J., & Buvens, I. (2009). Linking the policymaking capacities of schools and the quality of school self-evaluations: The view of school leaders. Educational Management Administration & Leadership, 37(5), 667–686. | DOI 10.1177/1741143209339653

[64] Vanhoof, J., & Van Petegem, P. (2007). Matching internal and external evaluation in an era of accountability and school development: lessons from a Flemish perspective. Studies in Educational Evaluation, 33(2), 101–119. | DOI 10.1016/j.stueduc.2007.04.001

[65] Wang, L., Beckett, G. H., & Brown, L. (2006). Controversies of standardized assessment in school accountability reform: A critical synthesis of multidisciplinary research evidence. Applied Measurement in Education, 19(4), 305–328. | DOI 10.1207/s15324818ame1904_5

[66] Wikeley, F. (1998). Dissemination of research as a tool for school improvement? School Leadership & Management, 18(1), 59–73. | DOI 10.1080/13632439869772

[67] Wiliam, D. (2010). Standardised testing and school accountability. Educational Psychologist, 45(2), 107–122. | DOI 10.1080/00461521003703060

[68] Wössmann, L. (2005). The effect heterogeneity of central examinations: Evidence from TIMSS, TIMSS-Repeat and PISA. Education Economics, 13(2), 143–169. | DOI 10.1080/09645290500031165