From signals to knowledge: A conceptual model for multimodal learning analytics

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science
Original languageEnglish
Pages (from-to)338-349
Number of pages12
JournalJournal of Computer Assisted Learning
Volume34
Issue number4
DOIs
Publication statusPublished - Aug 2018

Fingerprint

Learning systems
Sensors
learning
Terminology
Feedback
learning process
multimodality
Experiments
learning theory
science
field research
technical language
cognition
data analysis
emotion
interpretation
event
experiment

Keywords

  • learning analytics
  • machine learning
  • multimodal data
  • multimodality
  • sensors
  • social signal processing

Cite this

@article{abd8e4eaa9874fecb167d2c25c2f548a,
title = "From signals to knowledge: A conceptual model for multimodal learning analytics",
abstract = "Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science",
keywords = "learning analytics, machine learning, multimodal data, multimodality, sensors, social signal processing",
author = "{Di Mitri}, D. and Jan Schneider and M.M. Specht and H.J. Drachsler",
year = "2018",
month = "8",
doi = "10.1111/jcal.12288",
language = "English",
volume = "34",
pages = "338--349",
journal = "Journal of Computer Assisted Learning",
issn = "0266-4909",
publisher = "Wiley",
number = "4",

}

From signals to knowledge : A conceptual model for multimodal learning analytics. / Di Mitri, D.; Schneider , Jan; Specht, M.M.; Drachsler, H.J.

In: Journal of Computer Assisted Learning, Vol. 34, No. 4, 08.2018, p. 338-349.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - From signals to knowledge

T2 - A conceptual model for multimodal learning analytics

AU - Di Mitri, D.

AU - Schneider , Jan

AU - Specht, M.M.

AU - Drachsler, H.J.

PY - 2018/8

Y1 - 2018/8

N2 - Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science

AB - Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science

KW - learning analytics

KW - machine learning

KW - multimodal data

KW - multimodality

KW - sensors

KW - social signal processing

U2 - 10.1111/jcal.12288

DO - 10.1111/jcal.12288

M3 - Article

VL - 34

SP - 338

EP - 349

JO - Journal of Computer Assisted Learning

JF - Journal of Computer Assisted Learning

SN - 0266-4909

IS - 4

ER -