Augmented Reality and Affective Computing for Nonverbal Interaction Support of the Visually Impaired

Y.D. Iren; K. Shingjergji; C.C.J. Urlings; R. Klemke; Felix Böttger; Jelle Meindert Osinga; Sjoerd van de Goor; Damian Bustowski; Juliette Passariello-Jansen

doi:10.1109/VRW58643.2023.00080

Augmented Reality and Affective Computing for Nonverbal Interaction Support of the Visually Impaired

Y.D. Iren^*, K. Shingjergji, C.C.J. Urlings, R. Klemke, Felix Böttger, Jelle Meindert Osinga, Sjoerd van de Goor, Damian Bustowski, Juliette Passariello-Jansen

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference Article in proceeding › Academic › peer-review

Abstract

Nonverbal cues such as gestures and facial expressions are indispensable in human communication. However, such an essential aspect of social interactions is inaccessible to the sight impaired. This issue can be alleviated with the assistance of augmented reality and affective computing techniques embodied as wearable technology. Even though both augmented reality and affective computing have been studied comprehensively, the real-life deployment and utilization of these techniques are hindered by the limitations of wearable devices in terms of computational capabilities and battery life. This calls for a holistic approach to implementing lightweight and robust affective computing methods. In this study, we present a prototype that combines facial expression and gesture recognition that is optimized to function on battery-powered wearable devices. Additionally, the prototype embodies a haptic sleeve that communicates the detected facial expressions and gestures to the wearer.

Original language	English
Title of host publication	Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023
Publisher	IEEE
Pages	360- 363
Number of pages	4
ISBN (Electronic)	9798350348392
ISBN (Print)	9798350348408
DOIs	https://doi.org/10.1109/VRW58643.2023.00080
Publication status	Published - 25 Mar 2023
Event	IEEE VR 2023 Workshop on Multi-modal Affective and Social Behavior Analysis and Synthesis in Extended Reality - Online, Shanghai, China Duration: 25 Mar 2023 → 25 Mar 2023 https://sites.google.com/view/massxrworkshop2023/main

Workshop

Workshop	IEEE VR 2023 Workshop on Multi-modal Affective and Social Behavior Analysis and Synthesis in Extended Reality
Abbreviated title	IEEE-MASSXR
Country/Territory	China
City	Shanghai
Period	25/03/23 → 25/03/23
Internet address	https://sites.google.com/view/massxrworkshop2023/main

Keywords

Affective computing
augmented reality
human computer interaction
CCS [Human-centered computing]: Human computer interaction (HCI)
Computer vision
Accessibility technologies
CCS [Human-centered computing]: Accessibility
Interaction techniques CCS [Computing methodologies]: Artificial intelligence

Access to Document

10.1109/VRW58643.2023.00080

Cite this

Iren, Y. D., Shingjergji, K., Urlings, C. C. J., Klemke, R., Böttger, F., Meindert Osinga, J., van de Goor, S., Bustowski, D., & Passariello-Jansen, J. (2023). Augmented Reality and Affective Computing for Nonverbal Interaction Support of the Visually Impaired. In Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023 (pp. 360- 363). IEEE. https://doi.org/10.1109/VRW58643.2023.00080

@inproceedings{676f244311c046d59242caf00b502bd3,

title = "Augmented Reality and Affective Computing for Nonverbal Interaction Support of the Visually Impaired",

abstract = "Nonverbal cues such as gestures and facial expressions are indispensable in human communication. However, such an essential aspect of social interactions is inaccessible to the sight impaired. This issue can be alleviated with the assistance of augmented reality and affective computing techniques embodied as wearable technology. Even though both augmented reality and affective computing have been studied comprehensively, the real-life deployment and utilization of these techniques are hindered by the limitations of wearable devices in terms of computational capabilities and battery life. This calls for a holistic approach to implementing lightweight and robust affective computing methods. In this study, we present a prototype that combines facial expression and gesture recognition that is optimized to function on battery-powered wearable devices. Additionally, the prototype embodies a haptic sleeve that communicates the detected facial expressions and gestures to the wearer.",

keywords = "Affective computing, augmented reality, human computer interaction, CCS [Human-centered computing]: Human computer interaction (HCI), Computer vision, Accessibility technologies, CCS [Human-centered computing]: Accessibility, Interaction techniques CCS [Computing methodologies]: Artificial intelligence",

author = "Y.D. Iren and K. Shingjergji and C.C.J. Urlings and R. Klemke and Felix B{\"o}ttger and {Meindert Osinga}, Jelle and {van de Goor}, Sjoerd and Damian Bustowski and Juliette Passariello-Jansen",

year = "2023",

month = mar,

day = "25",

doi = "10.1109/VRW58643.2023.00080",

language = "English",

isbn = "9798350348408",

pages = "360-- 363",

booktitle = "Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023",

publisher = "IEEE",

address = "United States",

note = "IEEE VR 2023 Workshop on Multi-modal Affective and Social Behavior Analysis and Synthesis in Extended Reality, IEEE-MASSXR ; Conference date: 25-03-2023 Through 25-03-2023",

url = "https://sites.google.com/view/massxrworkshop2023/main",

}

Iren, YD , Shingjergji, K , Urlings, CCJ , Klemke, R, Böttger, F, Meindert Osinga, J, van de Goor, S, Bustowski, D & Passariello-Jansen, J 2023, Augmented Reality and Affective Computing for Nonverbal Interaction Support of the Visually Impaired. in Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. IEEE, pp. 360- 363, IEEE VR 2023 Workshop on Multi-modal Affective and Social Behavior Analysis and Synthesis in Extended Reality, Shanghai, China, 25/03/23. https://doi.org/10.1109/VRW58643.2023.00080

Augmented Reality and Affective Computing for Nonverbal Interaction Support of the Visually Impaired. / Iren, Y.D.; Shingjergji, K.; Urlings, C.C.J. et al.
Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. IEEE, 2023. p. 360- 363.

Research output: Chapter in Book/Report/Conference proceeding › Conference Article in proceeding › Academic › peer-review

TY - GEN

T1 - Augmented Reality and Affective Computing for Nonverbal Interaction Support of the Visually Impaired

AU - Iren, Y.D.

AU - Shingjergji, K.

AU - Urlings, C.C.J.

AU - Klemke, R.

AU - Böttger, Felix

AU - Meindert Osinga, Jelle

AU - van de Goor, Sjoerd

AU - Bustowski, Damian

AU - Passariello-Jansen, Juliette

PY - 2023/3/25

Y1 - 2023/3/25

N2 - Nonverbal cues such as gestures and facial expressions are indispensable in human communication. However, such an essential aspect of social interactions is inaccessible to the sight impaired. This issue can be alleviated with the assistance of augmented reality and affective computing techniques embodied as wearable technology. Even though both augmented reality and affective computing have been studied comprehensively, the real-life deployment and utilization of these techniques are hindered by the limitations of wearable devices in terms of computational capabilities and battery life. This calls for a holistic approach to implementing lightweight and robust affective computing methods. In this study, we present a prototype that combines facial expression and gesture recognition that is optimized to function on battery-powered wearable devices. Additionally, the prototype embodies a haptic sleeve that communicates the detected facial expressions and gestures to the wearer.

AB - Nonverbal cues such as gestures and facial expressions are indispensable in human communication. However, such an essential aspect of social interactions is inaccessible to the sight impaired. This issue can be alleviated with the assistance of augmented reality and affective computing techniques embodied as wearable technology. Even though both augmented reality and affective computing have been studied comprehensively, the real-life deployment and utilization of these techniques are hindered by the limitations of wearable devices in terms of computational capabilities and battery life. This calls for a holistic approach to implementing lightweight and robust affective computing methods. In this study, we present a prototype that combines facial expression and gesture recognition that is optimized to function on battery-powered wearable devices. Additionally, the prototype embodies a haptic sleeve that communicates the detected facial expressions and gestures to the wearer.

KW - Affective computing

KW - augmented reality

KW - human computer interaction

KW - CCS [Human-centered computing]: Human computer interaction (HCI)

KW - Computer vision

KW - Accessibility technologies

KW - CCS [Human-centered computing]: Accessibility

KW - Interaction techniques CCS [Computing methodologies]: Artificial intelligence

U2 - 10.1109/VRW58643.2023.00080

DO - 10.1109/VRW58643.2023.00080

M3 - Conference Article in proceeding

SN - 9798350348408

SP - 360

EP - 363

BT - Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

PB - IEEE

T2 - IEEE VR 2023 Workshop on Multi-modal Affective and Social Behavior Analysis and Synthesis in Extended Reality

Y2 - 25 March 2023 through 25 March 2023

ER -

Augmented Reality and Affective Computing for Nonverbal Interaction Support of the Visually Impaired

Abstract

Workshop

Keywords

Access to Document

Fingerprint

Cite this