Making inferential leaps: Manipulation checks and the road towards strong inference

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Experiments in psychology often target hypothetical constructs to test some causal hypothesis or theory. In light of this goal, it is pertinent to use a manipulation that actually changes the focal hypothetical construct, and only that construct. In assessing whether such manipulation “success” can be assumed, researchers often include manipulation validity checks in their designs—a measure of the focal construct which should be responsive to the manipulation. One interpretation of a positive manipulation check is that it lends credence to a particular causal interpretation of a phenomenon. Scrutinizing the results of such manipulation checks supposedly enables a more stringent test of a causal hypothesis. This paper submits that manipulation checks do not improve our inferences to causal explanations, but may in practice result in weaker hypothesis tests. Rather than being useful, manipulation checks are at best uninformative, but more likely compromise the appraisal of a causal hypothesis. The second half of this paper advocates four methodological heuristics, offered as alternatives to manipulation validity checks, to more severely test causal hypotheses. The heuristics call for a burgeoning focus on (a) assessing the specificity of manipulations, (b) evaluating theoretical risk, (c) attempts to cast doubt on alternatives, and (d) appraising the relative merits of explanations. I conclude that rather than relying on manipulation checks as a ‘Band-Aid’ method to alleviate validity concerns, inferential rigor can be improved by virtue of these heuristics.
Original languageEnglish
Article number104251
Number of pages9
JournalJournal of Experimental Social Psychology
Volume98
DOIs
Publication statusPublished - Jan 2022

Keywords

  • Manipulation check
  • Validity
  • Causal explanation
  • Inference to the best explanation
  • Falsificationism
  • Hypothetico-deductive method
  • CONSTRUCT-VALIDITY
  • MOOD
  • PROGRESS
  • EXPLANATION
  • PSYCHOLOGY
  • ATTENTION

Fingerprint

Dive into the research topics of 'Making inferential leaps: Manipulation checks and the road towards strong inference'. Together they form a unique fingerprint.

Cite this