The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics

Maren Scheffel*, Hendrik Drachsler, Christian Toisoul, Stefaan Ternier, Marcus Specht

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

12 Downloads (Pure)

Abstract

While learning analytics (LA) is maturing from being a trend to being part of the institutional toolbox, the need for more empirical evidences about the effects for LA on the actual stakeholders, i.e. learners and teachers, is increasing. Within this paper we report about a further evaluation iteration of the Evaluation Framework for Learning Analytics (EFLA) that provides an efficient and effective measure to get insights into the application of LA in educational institutes. For this empirical study we have thus developed and implemented several LA widgets into a MOOC platform’s dashboard and evaluated these widgets using the EFLA as well as the framework itself using principal component and reliability analysis. The results show that the EFLA is able to measure differences between widget versions. Furthermore, they indicate that the framework is highly reliable after slightly adapting its dimensions.
Original languageEnglish
Title of host publicationData Driven Approaches in Digital Education
Subtitle of host publicationProceedings of 12th European Conference on Technology Enhanced Learning EC-TEL 2017
EditorsÉ. Lavoué, H. Drachsler, K. Verbert, J. Broisin, M. Pérez-Sanagustín
PublisherSpringer
Pages194-208
ISBN (Electronic)978-3-319-66610-5
ISBN (Print)978-3-319-66609-9
DOIs
Publication statusPublished - 1 Sep 2017
EventData Driven Approaches in Digital Education: 12th European Conference on Technology Enhanced Learning: EC-TEL - Tallinn, Estonia
Duration: 12 Sep 201715 Sep 2017
http://ectel2017.httc.de/index.php?id=777

Publication series

NameLecture Notes in Computer Science
PublisherSpringer
Volume10474
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceData Driven Approaches in Digital Education
CountryEstonia
CityTallinn
Period12/09/1715/09/17
Internet address

Fingerprint

Reliability analysis
Principal component analysis

Keywords

  • learning analytics
  • evaluation
  • validity
  • reliability

Cite this

Scheffel, M., Drachsler, H., Toisoul, C., Ternier, S., & Specht, M. (2017). The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. In É. Lavoué, H. Drachsler, K. Verbert, J. Broisin, & M. Pérez-Sanagustín (Eds.), Data Driven Approaches in Digital Education: Proceedings of 12th European Conference on Technology Enhanced Learning EC-TEL 2017 (pp. 194-208). (Lecture Notes in Computer Science; Vol. 10474). Springer. https://doi.org/10.1007/978-3-319-66610-5_15
Scheffel, Maren ; Drachsler, Hendrik ; Toisoul, Christian ; Ternier, Stefaan ; Specht, Marcus. / The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. Data Driven Approaches in Digital Education: Proceedings of 12th European Conference on Technology Enhanced Learning EC-TEL 2017. editor / É. Lavoué ; H. Drachsler ; K. Verbert ; J. Broisin ; M. Pérez-Sanagustín. Springer, 2017. pp. 194-208 (Lecture Notes in Computer Science).
@inproceedings{5dd305c7ef7c4bdc9a2b84b87b3a46d6,
title = "The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics",
abstract = "While learning analytics (LA) is maturing from being a trend to being part of the institutional toolbox, the need for more empirical evidences about the effects for LA on the actual stakeholders, i.e. learners and teachers, is increasing. Within this paper we report about a further evaluation iteration of the Evaluation Framework for Learning Analytics (EFLA) that provides an efficient and effective measure to get insights into the application of LA in educational institutes. For this empirical study we have thus developed and implemented several LA widgets into a MOOC platform’s dashboard and evaluated these widgets using the EFLA as well as the framework itself using principal component and reliability analysis. The results show that the EFLA is able to measure differences between widget versions. Furthermore, they indicate that the framework is highly reliable after slightly adapting its dimensions.",
keywords = "learning analytics, evaluation, validity, reliability",
author = "Maren Scheffel and Hendrik Drachsler and Christian Toisoul and Stefaan Ternier and Marcus Specht",
year = "2017",
month = "9",
day = "1",
doi = "10.1007/978-3-319-66610-5_15",
language = "English",
isbn = "978-3-319-66609-9",
series = "Lecture Notes in Computer Science",
publisher = "Springer",
pages = "194--208",
editor = "Lavou{\'e}, {{\'E}. } and Drachsler, {H. } and Verbert, { K. } and Broisin, {J. } and P{\'e}rez-Sanagust{\'i}n, {M. }",
booktitle = "Data Driven Approaches in Digital Education",

}

Scheffel, M, Drachsler, H, Toisoul, C, Ternier, S & Specht, M 2017, The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. in É Lavoué, H Drachsler, K Verbert, J Broisin & M Pérez-Sanagustín (eds), Data Driven Approaches in Digital Education: Proceedings of 12th European Conference on Technology Enhanced Learning EC-TEL 2017. Lecture Notes in Computer Science, vol. 10474, Springer, pp. 194-208, Data Driven Approaches in Digital Education, Tallinn, Estonia, 12/09/17. https://doi.org/10.1007/978-3-319-66610-5_15

The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. / Scheffel, Maren; Drachsler, Hendrik; Toisoul, Christian; Ternier, Stefaan; Specht, Marcus.

Data Driven Approaches in Digital Education: Proceedings of 12th European Conference on Technology Enhanced Learning EC-TEL 2017. ed. / É. Lavoué; H. Drachsler; K. Verbert; J. Broisin; M. Pérez-Sanagustín. Springer, 2017. p. 194-208 (Lecture Notes in Computer Science; Vol. 10474).

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

TY - GEN

T1 - The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics

AU - Scheffel, Maren

AU - Drachsler, Hendrik

AU - Toisoul, Christian

AU - Ternier, Stefaan

AU - Specht, Marcus

PY - 2017/9/1

Y1 - 2017/9/1

N2 - While learning analytics (LA) is maturing from being a trend to being part of the institutional toolbox, the need for more empirical evidences about the effects for LA on the actual stakeholders, i.e. learners and teachers, is increasing. Within this paper we report about a further evaluation iteration of the Evaluation Framework for Learning Analytics (EFLA) that provides an efficient and effective measure to get insights into the application of LA in educational institutes. For this empirical study we have thus developed and implemented several LA widgets into a MOOC platform’s dashboard and evaluated these widgets using the EFLA as well as the framework itself using principal component and reliability analysis. The results show that the EFLA is able to measure differences between widget versions. Furthermore, they indicate that the framework is highly reliable after slightly adapting its dimensions.

AB - While learning analytics (LA) is maturing from being a trend to being part of the institutional toolbox, the need for more empirical evidences about the effects for LA on the actual stakeholders, i.e. learners and teachers, is increasing. Within this paper we report about a further evaluation iteration of the Evaluation Framework for Learning Analytics (EFLA) that provides an efficient and effective measure to get insights into the application of LA in educational institutes. For this empirical study we have thus developed and implemented several LA widgets into a MOOC platform’s dashboard and evaluated these widgets using the EFLA as well as the framework itself using principal component and reliability analysis. The results show that the EFLA is able to measure differences between widget versions. Furthermore, they indicate that the framework is highly reliable after slightly adapting its dimensions.

KW - learning analytics

KW - evaluation

KW - validity

KW - reliability

U2 - 10.1007/978-3-319-66610-5_15

DO - 10.1007/978-3-319-66610-5_15

M3 - Conference article in proceeding

SN - 978-3-319-66609-9

T3 - Lecture Notes in Computer Science

SP - 194

EP - 208

BT - Data Driven Approaches in Digital Education

A2 - Lavoué, É.

A2 - Drachsler, H.

A2 - Verbert, K.

A2 - Broisin, J.

A2 - Pérez-Sanagustín, M.

PB - Springer

ER -

Scheffel M, Drachsler H, Toisoul C, Ternier S, Specht M. The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. In Lavoué É, Drachsler H, Verbert K, Broisin J, Pérez-Sanagustín M, editors, Data Driven Approaches in Digital Education: Proceedings of 12th European Conference on Technology Enhanced Learning EC-TEL 2017. Springer. 2017. p. 194-208. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-319-66610-5_15