Towards multimodal emotion recognition in E-learning environments

Kiavash Bahreini, Rob Nadolski, Wim Westera

Research output: Contribution to journalArticleAcademicpeer-review

15 Downloads (Pure)

Abstract

This paper presents a framework (FILTWAM (Framework for Improving Learning Through Webcams And Microphones)) for real-time emotion recognition in e-learning by using webcams. FILTWAM offers timely and relevant feedback based upon learner’s facial expressions and verbalizations. FILTWAM’s facial expression software module has been developed and tested in a proof-of-concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with 10 test persons. They received the same computer-based tasks in which each of them were requested 100 times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants’ recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners’ behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner’s emotional states.
Original languageEnglish
Pages (from-to)590-605
Number of pages16
JournalInteractive LearnIng Environments
Volume24
Issue number3
Early online date12 May 2014
DOIs
Publication statusPublished - 2016

Fingerprint

E-learning
Microphones
Face recognition
learning environment
emotion
facial expression
Feedback
electronic learning
expert
software
video
interpretation
human being
learning
time

Keywords

  • Learner support in serious games
  • human–computer interaction
  • multimodal emotion recognition
  • eal-time face emotion recognition
  • webcam
  • E-learning

Cite this

@article{c9ebd7c5ea4c47239af693763173a0b7,
title = "Towards multimodal emotion recognition in E-learning environments",
abstract = "This paper presents a framework (FILTWAM (Framework for Improving Learning Through Webcams And Microphones)) for real-time emotion recognition in e-learning by using webcams. FILTWAM offers timely and relevant feedback based upon learner’s facial expressions and verbalizations. FILTWAM’s facial expression software module has been developed and tested in a proof-of-concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with 10 test persons. They received the same computer-based tasks in which each of them were requested 100 times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants’ recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72{\%}. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners’ behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner’s emotional states.",
keywords = "Learner support in serious games, human–computer interaction, multimodal emotion recognition, eal-time face emotion recognition, webcam, E-learning",
author = "Kiavash Bahreini and Rob Nadolski and Wim Westera",
year = "2016",
doi = "10.1080/10494820.2014.908927",
language = "English",
volume = "24",
pages = "590--605",
journal = "Interactive LearnIng Environments",
issn = "1049-4820",
publisher = "Routledge",
number = "3",

}

Towards multimodal emotion recognition in E-learning environments. / Bahreini, Kiavash; Nadolski, Rob; Westera, Wim.

In: Interactive LearnIng Environments, Vol. 24, No. 3, 2016, p. 590-605.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Towards multimodal emotion recognition in E-learning environments

AU - Bahreini, Kiavash

AU - Nadolski, Rob

AU - Westera, Wim

PY - 2016

Y1 - 2016

N2 - This paper presents a framework (FILTWAM (Framework for Improving Learning Through Webcams And Microphones)) for real-time emotion recognition in e-learning by using webcams. FILTWAM offers timely and relevant feedback based upon learner’s facial expressions and verbalizations. FILTWAM’s facial expression software module has been developed and tested in a proof-of-concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with 10 test persons. They received the same computer-based tasks in which each of them were requested 100 times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants’ recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners’ behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner’s emotional states.

AB - This paper presents a framework (FILTWAM (Framework for Improving Learning Through Webcams And Microphones)) for real-time emotion recognition in e-learning by using webcams. FILTWAM offers timely and relevant feedback based upon learner’s facial expressions and verbalizations. FILTWAM’s facial expression software module has been developed and tested in a proof-of-concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with 10 test persons. They received the same computer-based tasks in which each of them were requested 100 times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants’ recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners’ behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner’s emotional states.

KW - Learner support in serious games

KW - human–computer interaction

KW - multimodal emotion recognition

KW - eal-time face emotion recognition

KW - webcam

KW - E-learning

U2 - 10.1080/10494820.2014.908927

DO - 10.1080/10494820.2014.908927

M3 - Article

VL - 24

SP - 590

EP - 605

JO - Interactive LearnIng Environments

JF - Interactive LearnIng Environments

SN - 1049-4820

IS - 3

ER -