Sense the Classroom: Using AI to Detect and Respond to Learning-Centered Affective States in Online Education

Online learning has become an essential part of modern education, offering flexibility,
inclusivity, and accessibility for learners regardless of geographic or physical
constraints. However, despite these advantages, online environments lack
the nonverbal cues inherent in face-to-face interactions, making it difficult for educators
to recognize students’ learning-centered affective states (LCAS). This
disconnect can hinder timely pedagogical interventions and reduce the effectiveness
of learning. Affective computing offers a compelling solution to this challenge
by leveraging artificial intelligence (AI) to recognize students’ LCAS, such
as boredom, confusion, frustration, engagement, and curiosity. Unlike conventional
emotion recognition systems that focus on basic emotions such as happiness
or anger, LCAS provides more meaningful indicators of learning processes.
However, most existing technologies fall short in capturing these nuanced states,
and often overlook ethical concerns such as users’ privacy.
This doctoral thesis responds to these challenges by developing privacy- preserving,
AI-driven methods to detect and communicate LCAS in online higher education.
Through an interdisciplinary, design-based research approach, the thesis
explores how these technologies can support teachers in making informed,
timely decisions that keep learning optimal. Structured across four parts and
six chapters, each study contributes to the overarching goal of designing, implementing,
and evaluating tools that recognise students’ affective experiences
visibly, responsibly, and effectively, in online classrooms.
Part I provides a systematic literature review of affective computing in online
higher education in the period 2019-2024. This review identifies several shortcomings:
insufficient attention to LCAS, limited empirical validation of emotion
detection models in educational settings, and a notable lack of ethical concerns.
It also reveals a disciplinary divide between educational and technological research,
underscoring the need for interdisciplinary collaboration.
Part II focuses on technological development. Chapter 2 introduces an AI
model capable of detecting facial expressions in the form of Action Units (AUs)
from webcam inputs. To collect high-quality data for model training, the study develops
“FaceGame”, a gamified web application where participants mimic emotions
and receive feedback, enhancing both engagement and data quality. Chapter
3 expands on this by embedding the AU detection model into a real-time,
privacy-preserving system called “StC-live.” The system processes facial data
locally in the browser, ensuring no images or videos are stored or transmitted.
Teachers receive anonymized, aggregated LCAS data through a dashboard, al-
lowing them to monitor classroom dynamics without violating student privacy.
Part III addresses the pedagogical dimension. Chapter 4 engages educators
in a co-design process to understand their needs and preferences for LCAS
feedback systems. Teachers prioritize five LCAS and emphasize the importance
of minimizing cognitive overload while retaining control over the feedback
they receive. Chapter 5 develops standardized educational videos designed to
evoke specific LCAS, based on literature-derived principles and expert feedback.
These stimuli are critical for studying LCAS reliably in experimental settings.
Part IV integrates the technological and educational elements. Chapter 6 investigates
the correlation between students’ self-reported LCAS, detected facial
expressions, and personal background (e.g., topic familiarity, interest). The findings
confirm that students with higher affinity and prior knowledge report more
engagement and curiosity, while those with less experience are more likely to
feel bored or confused. Weak but notable correlations are observed between
certain AUs and LCAS, highlighting both the potential and limitations of facial
expression models in real-world conditions.
The thesis concludes with a discussion of contributions, limitations, and future
directions. Key technological contributions include AU detection models,
a gamified data collection platform, and the privacy-conscious StC-live system.
Educationally, the thesis contributes a validated taxonomy of LCAS, standardized
elicitation materials, and insights into how background factors influence affective
states in learning. However, several challenges remain: integrating the
components into a production-ready tool, collecting more diverse training data,
improving model robustness under varied conditions, and assessing the educational
impact of these tools in live classrooms. Ethically, the research emphasizes
transparency, informed consent, and data privacy. All human-subject
studies were approved by an ethics board, and technologies were designed to
mitigate the risk of misuse. Nevertheless, evolving regulations, particularly the
EU’s AI Act, pose uncertainties about deploying such systems in real educational
settings. This doctoral work offers a foundational framework for advancing affective
computing in online higher education, combining technological innovation
with educational sensitivity and ethical responsibility. It lays the groundwork for
future research and development aimed at making virtual learning more responsive,
empathetic, and effective.