ChatGPT in Education: An Effect in Search of a Cause

Background: As researchers rush to investigate the potential of AI tools like ChatGPT to enhance learning, well- documentedpitfalls threaten the validity of this emerging research. Issues of media comparison research, where the confounding of instruc-tional methods and technological affordances is unrecognised, may render effects uninterpretable.Objectives: Using a recent meta-analysis by Deng et al. (Computers & Education, 227, 105224) as an example, we revisit keyinsights from the media/methods debate to highlight recurring conceptual challenges in ChatGPT efficacy studies.Methods: This conceptual article contrasts nascent ChatGPT research with the more established literature on IntelligentTutoring Systems to identify three non-negotiable considerations for interpretable effects: (1) descriptions of the precise natureof the experimental treatment and (2) the activities of the control group, as well as (3) outcome measures as valid indicators oflearning. To provide some initial evidence, we audited a subset of primary experiments included in Deng et al.'s meta-analysis,demonstrating that only a small minority of studies satisfied all three non-negotiable considerations.Results and Conclusions: Loosely defined treatments, mismatched or opaque controls, and outcome measures with unclearlinks to durable learning obscure causal claims of this emerging literature. Observed gains cannot, at this time, be confidentlyattributed to ChatGPT, and meta-analytics effect sizes may over- or understate its benefits. Progress, we argue, will require rig-orous designs, transparent reporting, and a critical stance toward “fast science.”