Sensemaking unraveled: how teachers process school performance feedback data
Roč.26,č.4(2021)
Studia paedagogica: Professional Decision-Making in Education
The present study investigates how Flemish middle school mathematics teachers make sense of schoolperformanfeedback data from low-stakes, external standardised tests. We take an in-depth look into the interpretive steps they take, based on a conceptual model that integrates intuitive and rational aspects of individual and collective sensemaking and empirical data collected in semi-structured interviews. We describe the nature of these sensemaking processes and consider the impact of influencing factors. Our findings demonstrate that the mere availability of school performance feedback data does not spontaneously spark sensemaking, nor does it necessarily lead to improvements in instructionalpractice. Teachers' sensemaking of schoolperformancefeedback data appears to be a largely intuitive process, grounded in external attributions and absent of triangulation. Challenges regarding expertise and lack of inquiry-based attitude and commitment result in superficial and often incorrect interpretations of the data that tend to remain uncorrected as teachers barely engage in collaborative professional dialogue about the data.
sensemaking; data-driven decision making; school performance feedback; intuition; rationality
[1] Abrams, L. M., Varier, D., & Mehdi, T. (2020). The intersection of school context and teachers' data use practice: Implications for an integrated approach to capacity building. Studies in Educational Evaluation, 69, 1–13 . https://doi.org/10.1016/j.stueduc.2020.100868 | DOI 10.1016/j.stueduc.2020.100868
[2] Alase, A. (2017). The Interpretative Phenomenological Analysis (IPA): A guide to a good qualitative research approach. International Journal of Education and Literacy Studies, 5(2), 9–19. https://doi.org/10.7575/aiac.ijels.v.5n.2p.9 | DOI 10.7575/aiac.ijels.v.5n.2p.9
[3] Amels, J., Krüger, M., Suhre, C., & van Veen, K. (2019). Impact of inquiry-based working on the capacity to change in primary education. Journal of Educational Change, 20(3), 351–374. https://doi.org/10.1007/s10833-019-09337-3 | DOI 10.1007/s10833-019-09337-3
[4] Amidu, A. R., Boyd, D., & Gobet, F. (2019). A study of the interplay between intuition and rationality in valuation decision making. Journal of Property Research, 36(4), 387–418. https://doi.org/10.1080/09599916.2019.1687572 | DOI 10.1080/09599916.2019.1687572
[5] Attfield, S., & Baber, C. (2017, June 20). Elaborating the frames of data-frame theory. 13th International Conference on Naturalistic Decision Making, Bath, United Kingdom.
[6] Attfield, S., Fields, B., & Baber, C. (2018). A resources model for distributed sensemaking. Cognition, Technology and Work, 20(4), 651–664. https://doi.org/10.1007/s10111-018-0529-4 | DOI 10.1007/s10111-018-0529-4
[7] Avby, G. (2015). Professional practice as processes of muddling through: A study of learning and sense making in social work. Vocations and Learning, 8(1), 95–113. https://doi.org/10.1007/s12186-014-9123-y | DOI 10.1007/s12186-014-9123-y
[8] Beck, J. S., & Nunnaley, D. (2021). A continuum of data literacy for teaching. Studies in Educational Evaluation, 69, 1–8. https://doi.org/10.1016/j.stueduc.2020.100871 | DOI 10.1016/j.stueduc.2020.100871
[9] Bertrand, M., & Marsh, J. A. (2015). Teachers' sensemaking of data and implications for equity. American Educational Research Journal, 52(5), 861–893. https://doi.org/10.3102/0002831215599251 | DOI 10.3102/0002831215599251
[10] Bolhuis, E., Schildkamp, K., & Voogt, J. (2016). Data-based decision making in teams: Enablers and barriers. Educational Research and Evaluation, 22(3–4), 213–233. https://doi.org/10.1080/13803611.2016.1247728 | DOI 10.1080/13803611.2016.1247728
[11] Brock, R. (2015). Intuition and insight: Two concepts that illuminate the tacit in science education. Studies in Science Education, 51(2), 127–167. https://doi.org/10.1080/03057267.2015.1049843 | DOI 10.1080/03057267.2015.1049843
[12] Calabretta, G., Gemser, G., & Wijnberg, N. M. (2017). The interplay between intuition and rationality in strategic decision making: A paradox perspective. Organization Studies, 38(3–4), 365–401. https://doi.org/10.1177/0170840616655483 | DOI 10.1177/0170840616655483
[13] Cameron, M., & Lovett, S. (2015). Sustaining the commitment and realising the potential of highly promising teachers. Teachers and Teaching: Theory and Practice, 21(2), 150–163. https://doi.org/10.1080/13540602.2014.928132 | DOI 10.1080/13540602.2014.928132
[14] Cecez-Kecmanovic, D. (2004). A sensemaking model of knowledge in organisations: A way of understanding knowledge management and the role of information technologies. Knowledge Management Research & Practice, 2(3), 155–168. https://doi.org/10.1057/palgrave.kmrp.8500040 | DOI 10.1057/palgrave.kmrp.8500040
[15] Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121–152. https://doi.org/10.1207/s15516709cog0502_2 | DOI 10.1207/s15516709cog0502_2
[16] Christman, J. B., Ebby, C., & Edmunds, K. (2016). Data use practices for improved mathematics teaching and learning: The importance of productive dissonance and recurring feedback cycles. Teachers College Record, 118(11), 1–32. | DOI 10.1177/016146811611801101
[17] Chudnoff, E. (2019). In search of intuition. Australasian Journal of Philosophy, 98(3), 465–480. https://doi.org/10.1080/00048402.2019.1658121 | DOI 10.1080/00048402.2019.1658121
[18] Coburn, C. E., & Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research & Perspective, 9(4), 173–206. https://doi.org/10.1080/15366367.2011.626729 | DOI 10.1080/15366367.2011.626729
[19] Cohen, L., Martin, S., McCulloch, G., O'Sullivan, C., Manion, L., Morrison, K., & Bell, R. (2011). Research methods in education (7th ed.). Routledge.
[20] Cook, L., & Gregory, M. (2019). Making sense of sensemaking: Conceptualising how child and family social workers process assessment information. Child Care in Practice, 26(2), 182–195. https://doi.org/10.1080/13575279.2019.1685458 | DOI 10.1080/13575279.2019.1685458
[21] Dane, E., & Pratt, M. G. (2007). Exploring intuition and its role in managerial decision making. Academy of Management Review, 32(1), 33–54. https://doi.org/10.5465/AMR.2007.23463682 | DOI 10.5465/amr.2007.23463682
[22] Datnow, A., & Park, V. (2018). Opening or closing doors for students? Equity and data use in schools. Journal of Educational Change, 19(2), 131–152. https://doi.org/10.1007/s10833-018-9323-6 | DOI 10.1007/s10833-018-9323-6
[23] Datnow, A., Park, V., & Kennedy-Lewis, B. (2012). High school teachers' use of data to inform instruction. Journal of Education for Students Placed at Risk, 17(4), 247–265. https://doi.org/10.1080/10824669.2012.718944 | DOI 10.1080/10824669.2012.718944
[24] Day, C., & Gu, Q. (2007). Variations in the conditions for teachers' professional learning and development: Sustaining commitment and effectiveness over a career. Oxford Review of Education, 33(4), 423–443. https://doi.org/10.1080/03054980701450746 | DOI 10.1080/03054980701450746
[25] Donche, V. (2015). Inleiding in methoden en technieken. Een open leerpakket [Introduction to methods and techniques. An open learning package]. University of Antwerp.
[26] Eccles, D. W., & Arsal, G. (2017). The think aloud method: What is it and how do I use it? Qualitative Research in Sport, Exercise and Health, 9(4), 514–531. https://doi.org/10.1080/2159676X.2017.1331501 | DOI 10.1080/2159676X.2017.1331501
[27] Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383. https://doi.org/10.2307/2666999 | DOI 10.2307/2666999
[28] Edmondson, A. C., & Lei, Z. (2014). Psychological safety: The history, renaissance, and future of an interpersonal construct. Annual Review of Organizational Psychology and Organizational Behavior, 1(1), 23–43. https://doi.org/10.1146/annurev-orgpsych-031413-091305 | DOI 10.1146/annurev-orgpsych-031413-091305
[29] Fransson, G., & Frelin, A. (2016). Highly committed teachers: What makes them tick? A study of sustained commitment. Teachers and Teaching: Theory and Practice, 22(8), 896–912. https://doi.org/10.1080/13540602.2016.1201469 | DOI 10.1080/13540602.2016.1201469
[30] Gephart, R. P., Topal, C., & Zhang, Z. (2010). Future-oriented sensemaking: Temporalities and institutional legitimation. In T. Hernes & S. Maitlis (Eds.), Process, sensemaking and organizing (pp. 275–312). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199594566.003.0013 | DOI 10.1093/acprof:oso/9780199594566.003.0013
[31] Gergen, K. J., Gergen, M. M., & Barrett, F. J. (2004). Dialogue: Life and death of the organization. In D. Grant, C. Oswick, & L. Putnam (Eds.), The Sage handbook of organizational discourse (pp. 39–60). Sage. https://doi.org/10.4135/9781848608122.n2 | DOI 10.4135/9781848608122.n2
[32] Hill, R. C., & Levenhagen, M. (1995). Metaphors and mental models: Sensemaking and sense giving in innovative and entrepreneurial activities. Journal of Management, 21(6), 1057–1074. https://doi.org/10.1177/014920639502100603 | DOI 10.1177/014920639502100603
[33] Horn, I. S., & Little, J. W. (2010). Attending to problems of practice: Routines and resources for professional learning in teachers' workplace interactions. American Educational Research Journal, 47(1), 181–217. https://doi.org/10.3102/0002831209345158 | DOI 10.3102/0002831209345158
[34] Jimerson, J. B. (2014). Thinking about data: Exploring the development of mental models for "data use" among teachers and school leaders. Studies in Educational Evaluation, 42, 5–14. https://doi.org/10.1016/j.stueduc.2013.10.010 | DOI 10.1016/j.stueduc.2013.10.010
[35] Kahneman, D. (2011). Thinking, fast and slow. Business Contact.
[36] Kahneman, D., & Frederick, S. (2005). A model of heuristic judgment. In K. Holyoak & B. Morrison (Eds.). Cambridge handbook of thinking and reasoning (pp. 267–293). Cambridge University Press.
[37] Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526. https://doi.org/10.1037/a0016755 | DOI 10.1037/a0016755
[38] Kahneman, D., & Tversky, A. (1979). Prospect Theory: An analysis of decision under risk. Econometrica, 47(2), 263–291. | DOI 10.2307/1914185
[39] Klein, G., Phillips, J. K., Rall, E. L., & Peluso, D. A. (2007). A data-frame theory of sensemaking. In Expertise out of context: Proceedings of the sixth international conference on naturalistic decision making (pp. 113–155). Lawrence Erlbaum Associates.
[40] Krüger, M. L. (2018). Onderzoeksmatig leidinggeven: Handreiking voor schoolleiders en bestuurders [Inquiry-based leadership: Guide for school leaders and administrators]. ACCO.
[41] Krüger, M. L. (2010). De invloed van schoolleiderschap op het onderzoeksmatig handelen van leraren in veranderingsprocessen [The impact of school leadership on teachers' inquiry-based work in change processes]. Hogeschool van Amsterdam.
[42] Kudesia, R. S. (2017). Organizational sensemaking. In Oxford Research Encyclopedia of Psychology (pp. 1–42). Oxford University Press. https://doi.org/10.1093/acrefore/9780190236557.013.78 | DOI 10.1093/acrefore/9780190236557.013.78
[43] Lai, M. K., Wilson, A., McNaughton, S., & Hsiao, S. (2014). Improving achievement in secondary schools: Impact of a literacy project on reading comprehension and secondary school qualifications. Reading Research Quarterly, 49(3), 305–334. https://doi.org/10.1002/rrq.73 | DOI 10.1002/rrq.73
[44] Maitlis, S. (2005). The social processes of organizational sensemaking. Academy of Management Journal, 48(1), 21–49. https://doi.org/10.5465/AMJ.2005.15993111 | DOI 10.5465/amj.2005.15993111
[45] Maitlis, S., & Christianson, M. (2014). Sensemaking in organizations: Taking stock and moving forward. The Academy of Management Annals, 8(1), 57–125. https://doi.org/10.1080/19416520.2014.873177 | DOI 10.5465/19416520.2014.873177
[46] Maitlis, S., Vogus, T. J., & Lawrence, T. B. (2013). Sensemaking and emotion in organizations. Organizational Psychology Review, 3(3), 222–247. https://doi.org/10.1177/2041386613489062 | DOI 10.1177/2041386613489062
[47] Mandinach, E. B., & Gummer, E. S. (2016). What does it mean for teachers to be data literate: Laying out the skills, knowledge, and dispositions. Teaching and Teacher Education, 60, 366–376. https://doi.org/10.1016/j.tate.2016.07.011 | DOI 10.1016/j.tate.2016.07.011
[48] Mandinach, E. B., & Schildkamp, K. (2021). Misconceptions about data-based decision making in education: An exploration of the literature. Studies in Educational Evaluation, 69, 1–10. https://doi.org/10.1016/j.stueduc.2020.100842 | DOI 10.1016/j.stueduc.2020.100842
[49] Mausethagen, S., Prøitz, T. S., & Skedsmo, G. (2019). School leadership in data use practices: Collegial and consensus-oriented. Educational Research, 61(1), 70–86. https://doi.org/10.1080/00131881.2018.1561201 | DOI 10.1080/00131881.2018.1561201
[50] Nonaka, I., & Takeuchi, H. (1995). The knowledge-creating company. Oxford University Press.
[51] Prenger, R., & Schildkamp, K. (2018). Data-based decision making for teacher and student learning: A psychological perspective on the role of the teacher. Educational Psychology, 38(6), 734–752. https://doi.org/10.1080/01443410.2018.1426834 | DOI 10.1080/01443410.2018.1426834
[52] Roesch-Marsh, A. (2018). Professional relationships and decision making in social work: Lessons from a Scottish case study of secure accommodation decision making. Qualitative Social Work, 17(3), 405–422. https://doi.org/10.1177/1473325016680285 | DOI 10.1177/1473325016680285
[53] Salas, E., Rosen, M. A., & DiazGranados, D. (2010). Expertise-based intuition and decision making in organizations. Journal of Management, 36(4), 941–973. https://doi.org/10.1177/0149206309350084 | DOI 10.1177/0149206309350084
[54] Sammons, P., Day, C., Kington, A., Gu, Q., Stobart, G., & Smees, R. (2007). Exploring variations in teachers' work, lives and their effects on pupils: Key findings and implications from a longitudinal mixed-method study. British Educational Research Journal, 33(5), 681–701. https://doi.org/10.1080/01411920701582264 | DOI 10.1080/01411920701582264
[55] Schildkamp, K. (2019). Data-based decision-making for school improvement: Research insights and gaps. Educational Research, 61(3), 257–273. https://doi.org/10.1080/00131881.2019.1625716 | DOI 10.1080/00131881.2019.1625716
[56] Schildkamp, K., Lai, M. K., & Earl, L. (Eds.) (2013). Data-based decision making in education. Challenges and Opportunities. Springer.
[57] Schildkamp, K., Smit, M., & Blossing, U. (2019). Professional development in the use of data: From data to knowledge in data teams. Scandinavian Journal of Educational Research, 63(3), 393–411. https://doi.org/10.1080/00313831.2017.1376350 | DOI 10.1080/00313831.2017.1376350
[58] Schildkamp, K., & Teddlie, C. (2008). School performance feedback systems in the USA and in The Netherlands: A comparison. Educational Research and Evaluation, 14(3), 255–282. https://doi.org/10.1080/13803610802048874 | DOI 10.1080/13803610802048874
[59] Shleifer, A. (2012). Psychologists at the gate: A review of Daniel Kahneman's Thinking, Fast and Slow. Journal of Economic Literature, 50(4), 1080-91. http://doi.org/10.1257/jel.50.4.1080 | DOI 10.1257/jel.50.4.1080
[60] Sinclair, M. (2010). Misconceptions about intuition. Psychological Inquiry, 21(4), 378–386. https://doi.org/10.1080/1047840X.2010.523874 | DOI 10.1080/1047840X.2010.523874
[61] Spillane, J. P. (2012). Data in practice: Conceptualizing the data-based decision-making phenomena. American Journal of Education, 118(2), 113–141. https://doi.org/10.1086/663283 | DOI 10.1086/663283
[62] Spillane, J. P., & Miele, D. B. (2007). Evidence in practice: A framing of the terrain. Teachers College Record, 109(13), 46-73. | DOI 10.1177/016146810710901303
[63] Spillane, J. P., Reiser, B. J., & Reimer, T. (2002). Policy implementation and cognition: Reframing and refocusing implementation research. Review of Educational Research, 72(3), 387–431. https://doi.org/10.3102/00346543072003387 | DOI 10.3102/00346543072003387
[64] Tsoukas, H. (2009). A dialogical approach to the creation of new knowledge in organizations. Organization Science, 20(6), 941–957. https://doi.org/10.1287/orsc.1090.0435 | DOI 10.1287/orsc.1090.0435
[65] Uiterwijk-Luijk, L., Krüger, M., Zijlstra, B., & Volman, M. (2017). The relationship between psychological factors and inquiry-based working by primary school teachers. Educational Studies, 43(2), 147–164. https://doi.org/10.1080/03055698.2016.1248901 | DOI 10.1080/03055698.2016.1248901
[66] Van Gasse, R., Vanhoof, J., Mahieu, P., & Van Petegem, P. (2015). Informatiegebruik door schoolleiders en leerkrachten [Information use by school leaders and teachers]. Garant.
[67] Van Gasse, R., Vanhoof, J., & de Vos, W. (2014). Informatiegebruik in verantwoordingsgerichte en schoolontwikkelingsgerichte onderwijssystemen: Een Nederlands-Vlaams perspectief [Information use in accountability oriented and school development oriented education systems: A Dutch-Flemish perspective]. Pedagogiek, 34(2), 84–106. | DOI 10.5117/PED2014.2.GASS
[68] Van Gasse, R., Vanlommel, K., Vanhoof, J., & Van Petegem, P. (2017). The impact of collaboration on teachers' individual data use. School Effectiveness and School Improvement, 28(3), 489–504. https://doi.org/10.1080/09243453.2017.1321555 | DOI 10.1080/09243453.2017.1321555
[69] Vanhoof, J., Verhaeghe, G., Verhaeghe, J. P., Valcke, M., & Van Petegem, P. (2011). The influence of competences and support on school performance feedback use. Educational Studies, 37(2), 141–154. https://doi.org/10.1080/03055698.2010.482771 | DOI 10.1080/03055698.2010.482771
[70] Vanlommel, K., & Schildkamp, K. (2019). How do teachers make sense of data in the context of high-stakes decision making? American Educational Research Journal, 56(3), 792–821. https://doi.org/10.3102/0002831218803891 | DOI 10.3102/0002831218803891
[71] Vanlommel, K., Van Gasse, R., Vanhoof, J., & Van Petegem, P. (2018). Teachers' high-stakes decision making. How teaching approaches affect rational and intuitive data collection. Teaching and Teacher Education, 71, 108–119. https://doi.org/10.1016/j.tate.2017.12.011 | DOI 10.1016/j.tate.2017.12.011
[72] Vanlommel, K., Van Gasse, R., Vanhoof, J., & Van Petegem, P. (2017). Teachers' decisionmaking: Data based or intuition driven? International Journal of Educational Research, 83, 75–83. https://doi.org/10.1016/j.ijer.2017.02.013 | DOI 10.1016/j.ijer.2017.02.013
[73] Verhaeghe, G., Vanhoof, J., Valcke, M., & Van Petegem, P. (2010). Using school performance feedback: Perceptions of primary school principals. School Effectiveness and School Improvement, 21(2), 167–188. https://doi.org/10.1080/09243450903396005 | DOI 10.1080/09243450903396005
[74] Visscher, A. J., & Coe, R. (2003). School performance feedback systems: Conceptualisation, analysis, and reflection. School Effectiveness and School Improvement, 14(3), 321–349. https://doi.org/10.1076/sesi.14.3.321.15842 | DOI 10.1076/sesi.14.3.321.15842
[75] Weick, K. E. (1995). Sensemaking in organizations. Sage Publications.
[76] Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (2005). Organizing and the process of sensemaking. Organization Science, 16(4), 409–421. https://doi.org/10.1287/orsc.1050.0133 | DOI 10.1287/orsc.1050.0133
[77] Whittaker, A. (2018). How do child-protection practitioners make decisions in real-life situations? Lessons from the psychology of decision-making. British Journal of Social Work, 48(7), 1967–1984. | DOI 10.1093/bjsw/bcx145
[78] Zhang, P., & Soergel, D. (2016). Process patterns and conceptual changes in knowledge representations during information seeking and sensemaking: A qualitative user study. Journal of Information Science, 42(1), 59–78. https://doi.org/10.1177/0165551515615834 | DOI 10.1177/0165551515615834
[79] Zhang, P., & Soergel, D. (2019). Cognitive mechanisms in sensemaking: A qualitative user study. Journal of the Association for Information Science and Technology, 71(2), 158–171. https://doi.org/10.1002/asi.24221 | DOI 10.1002/asi.24221
