Data Fusion for Real-time Multimodal Emotion Recognition through Webcams and Microphones in E-Learning

Kiavash Bahreini, Rob Nadolski, Wim Westera

Research output: Contribution to journalArticleAcademicpeer-review

1 Downloads (Pure)

Abstract

This paper describes the validation study of our software that uses combined webcam and microphone data for real-time, continuous, unobtrusive emotion recognition as part of our FILTWAM framework. FILTWAM aims at deploying a real time multimodal emotion recognition method for providing more adequate feedback to the learners through an online communication skills training. Herein, timely feedback is needed that reflects on their shown intended emotions and which is also useful to increase learners’ awareness of their own behaviour. At least, a reliable and valid software interpretation of performed face and voice emotions is needed to warrant such adequate feedback. This validation study therefore calibrates our software. The study uses a multimodal fusion method. Twelve test persons performed computer-based tasks in which they were asked to mimic specific facial and vocal emotions. All test persons’ behaviour was recorded on video and two raters independently scored the showed emotions, which were contrasted with the software recognition outcomes. A hybrid method for multimodal fusion of our multimodal software shows accuracy between 96.1% and 98.6% for the best-chosen WEKA classifiers over predicted emotions. The software fulfils its requirements of real-time data interpretation and reliable results.
Original languageEnglish
Pages (from-to)415-430
JournalInternational Journal of Human-Computer Interaction
Volume32
Issue number5
DOIs
Publication statusPublished - 2 Mar 2016

Fingerprint

E-learning
Data fusion
Microphones
Feedback
Classifiers
Communication

Keywords

  • Hybrid Data Fusion
  • Multimodal Emotion Recognition
  • Emotion Detection
  • Real-time Software Development
  • Software Development
  • Data Mining
  • WEKA Classifiers
  • Machine Learning
  • Webcam
  • Microphone

Cite this

@article{e7c1944bb84a437a8cfb1419139c60c7,
title = "Data Fusion for Real-time Multimodal Emotion Recognition through Webcams and Microphones in E-Learning",
abstract = "This paper describes the validation study of our software that uses combined webcam and microphone data for real-time, continuous, unobtrusive emotion recognition as part of our FILTWAM framework. FILTWAM aims at deploying a real time multimodal emotion recognition method for providing more adequate feedback to the learners through an online communication skills training. Herein, timely feedback is needed that reflects on their shown intended emotions and which is also useful to increase learners’ awareness of their own behaviour. At least, a reliable and valid software interpretation of performed face and voice emotions is needed to warrant such adequate feedback. This validation study therefore calibrates our software. The study uses a multimodal fusion method. Twelve test persons performed computer-based tasks in which they were asked to mimic specific facial and vocal emotions. All test persons’ behaviour was recorded on video and two raters independently scored the showed emotions, which were contrasted with the software recognition outcomes. A hybrid method for multimodal fusion of our multimodal software shows accuracy between 96.1{\%} and 98.6{\%} for the best-chosen WEKA classifiers over predicted emotions. The software fulfils its requirements of real-time data interpretation and reliable results.",
keywords = "Hybrid Data Fusion, Multimodal Emotion Recognition, Emotion Detection, Real-time Software Development, Software Development, Data Mining, WEKA Classifiers, Machine Learning, Webcam, Microphone",
author = "Kiavash Bahreini and Rob Nadolski and Wim Westera",
year = "2016",
month = "3",
day = "2",
doi = "10.1080/10447318.2016.1159799",
language = "English",
volume = "32",
pages = "415--430",
journal = "International Journal of Human-Computer Interaction",
issn = "1532-7590",
publisher = "Taylor & Francis Group",
number = "5",

}

Data Fusion for Real-time Multimodal Emotion Recognition through Webcams and Microphones in E-Learning. / Bahreini, Kiavash; Nadolski, Rob; Westera, Wim.

In: International Journal of Human-Computer Interaction, Vol. 32, No. 5, 02.03.2016, p. 415-430.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Data Fusion for Real-time Multimodal Emotion Recognition through Webcams and Microphones in E-Learning

AU - Bahreini, Kiavash

AU - Nadolski, Rob

AU - Westera, Wim

PY - 2016/3/2

Y1 - 2016/3/2

N2 - This paper describes the validation study of our software that uses combined webcam and microphone data for real-time, continuous, unobtrusive emotion recognition as part of our FILTWAM framework. FILTWAM aims at deploying a real time multimodal emotion recognition method for providing more adequate feedback to the learners through an online communication skills training. Herein, timely feedback is needed that reflects on their shown intended emotions and which is also useful to increase learners’ awareness of their own behaviour. At least, a reliable and valid software interpretation of performed face and voice emotions is needed to warrant such adequate feedback. This validation study therefore calibrates our software. The study uses a multimodal fusion method. Twelve test persons performed computer-based tasks in which they were asked to mimic specific facial and vocal emotions. All test persons’ behaviour was recorded on video and two raters independently scored the showed emotions, which were contrasted with the software recognition outcomes. A hybrid method for multimodal fusion of our multimodal software shows accuracy between 96.1% and 98.6% for the best-chosen WEKA classifiers over predicted emotions. The software fulfils its requirements of real-time data interpretation and reliable results.

AB - This paper describes the validation study of our software that uses combined webcam and microphone data for real-time, continuous, unobtrusive emotion recognition as part of our FILTWAM framework. FILTWAM aims at deploying a real time multimodal emotion recognition method for providing more adequate feedback to the learners through an online communication skills training. Herein, timely feedback is needed that reflects on their shown intended emotions and which is also useful to increase learners’ awareness of their own behaviour. At least, a reliable and valid software interpretation of performed face and voice emotions is needed to warrant such adequate feedback. This validation study therefore calibrates our software. The study uses a multimodal fusion method. Twelve test persons performed computer-based tasks in which they were asked to mimic specific facial and vocal emotions. All test persons’ behaviour was recorded on video and two raters independently scored the showed emotions, which were contrasted with the software recognition outcomes. A hybrid method for multimodal fusion of our multimodal software shows accuracy between 96.1% and 98.6% for the best-chosen WEKA classifiers over predicted emotions. The software fulfils its requirements of real-time data interpretation and reliable results.

KW - Hybrid Data Fusion

KW - Multimodal Emotion Recognition

KW - Emotion Detection

KW - Real-time Software Development

KW - Software Development

KW - Data Mining

KW - WEKA Classifiers

KW - Machine Learning

KW - Webcam

KW - Microphone

U2 - 10.1080/10447318.2016.1159799

DO - 10.1080/10447318.2016.1159799

M3 - Article

VL - 32

SP - 415

EP - 430

JO - International Journal of Human-Computer Interaction

JF - International Journal of Human-Computer Interaction

SN - 1532-7590

IS - 5

ER -