In the Eye of the Beholder…

: A mixed methods research on interpretation, appreciation and use of the dashboard ‘Category Analysis’ for mathematics in Cito LOVS computer program.

Abstract

Schools internationally and also in the Netherlands are stimulated to use data to enhance their education: ‘data driven decision making’ (DDDM). By using assessment data in a formative way, adjustment of the teaching process can be achieved. Schools for primary education in the Netherlands are obliged to use a pupil monitoring system. The tests from Cito are used in the vast majority of the schools. Most schools also use the computer program ‘Cito LOVS’ to analyse the gathered data from the tests. One of the possible outputs in this program, a dashboard, is the ‘category analysis’ for mathematics. This ‘category analysis’ gives the user the opportunity to examine if a pupil performs differently in certain domains of mathematics than would be expected based on the ability level the pupil achieved on the entire test. To use the information of assessments in a meaningful way, the correct interpretation of the dashboard is thus vital. The study presented in this thesis has been executed in commission of Cito. The dashboard ‘category analysis’ (CA) has been in use for several years but has not been evaluated in a systematic way with users. This study aimed to research the intended interpretation of CA, the way users interpret and value CA, and to investigate if design enhancements can improve interpretation CA by its users. The study used a mixed methods design. Cito experts were interviewed to determine the way CA was intended to be interpreted. Based on these interviews a redesign was performed. Two focus groups, each consisting of four teaching professionals in primary education, were conducted. The focus groups were shown original and redesigned dashboard examples and asked how they would interpret them. Results were used to construct a survey. The survey was completed by 278 professionals in primary education. The survey consisted of three sets of original and redesigned CA dashboards and four depicted elements of the CA dashboard. Respondents were asked multiple response questions about interpretation and multiple-choice question about knowledge of CA elements. Another part of the survey consisted of a Dutch translation of the EFLA questionnaire (Scheffel, Drachsler, Toisoul, Ternier, & Specht, 2017), to evaluate CA. The actual interpretation of CA by users differs dramatically from the intentions as formulated by the Cito Experts. Especially the interpretation of pupil profiles that are indicated ‘not prominent’ cause problems. According to Cito experts these profiles should not be investigated further. Nevertheless, focus groups and over 90% of respondents in the survey indicate they would investigate these profiles in spite of this indication. Redesign of the CA dashboard did not lead to improved interpretation by users. This study suggests the use of CA leads to over-signalling of problems and might even lead to giving wrong or unnecessary remediation to the test taker. A radical redesign of CA without the use of graphics could be a next step to investigate. This study also supports the view that test reports and dashboards should be field tested, comparable to the testing of test/quiz/survey items, during the design phase and before distributing and implementing them in a running system.

Date of Award	13 Oct 2019
Original language	English
Supervisor	Maren Scheffel (Supervisor)