Lecturers’ Beliefs and Needs concerning Embedded Assessment in Online Higher Education:an Exploratory Study.

Abstract

Background. Recent development towards data driven education have led the Open University (OUNL) to explore and elaborate technical possibilities of its digital learning environment and wider data infrastructure. Complementary to these technical explorations, this study investigates educational affordances of data driven education, more particularly, embedded assessment: continuous, nearly unnoticed data collection while learners execute learning activities. This kind of assessment is considered to raise assessment validity, especially with regard to highly complex skills, and has a combined formative-summative purpose.
Aim. This thesis research is a first exploration of OUNL’s lecturers’ beliefs and needs considering embedded assessment in their own educational practice. Based on this, tentative design principles for embedded assessment in the particular educational context are formulated.
Participants, procedure, design. Nine lecturers of the OUNL master of Educational Sciences, varying in age (M = 44.67, SD = 12.18) and professional experience (less than five to more than 20 years), consented to participate in individual open-ended interviews. This qualitative exploratory study is carried out as a first step (systematic analysis) within a design-based research design.
Measures. The materials used during the interviews include an interview protocol and an introductory video explaining the concept of embedded assessment to participants. Before use, these materials were pilot tested and refined.
Results. Participants’ general reflections on embedded assessment were positive but with some reservations regarding the use of embedded assessment for complex skills as well as for summative purposes. Embedded assessment was expected to increase the availability and effectiveness of (automated) feedback, to facilitate agile and differentiated instructional adaptations, and to foster more reliable summative decision making. However, lecturers expressed multiple concerns regarding assessment validity. Anticipated effects of embedded assessment implementation on student experience and self-regulation and on lecturer work load were not without ambiguity. All lecturers perceived concrete opportunities for future use of embedded assessment in their own courses, mainly in support of formative assessment purposes. Perceived opportunities focus on specific learning objectives reflected in products the student creates as well as on student self-regulation and collaborative learning.
Conclusion. Based on lecturers’ views and suggestions, tentative design principles for embedded assessment are formulated. Considering possible future implementation of embedded assessment in this particular educational program, further requirement elicitation and thorough research regarding underlying learning progressions are necessary. With regard to the general exploration of lecturers’ beliefs and needs regarding embedded assessment, future research could focus on different educational programs and universities. The developed embedded assessment framework is found to be useful to introduce the concept to participants, although explicating the possibility to provide embedded assessment information directly to students seems necessary. The frameworks clarity to other potential user groups should be tested before use.

Date of Award	25 Jan 2020
Original language	English
Supervisor	Jose Janssen (Supervisor)