Matthias Siemer
Abstract
The central question of the paper is: When should stimuli be treated as random effects in random- or mixed-effects ANOVA? The conceptual starting point is the view of significance testing in experimental research as following the logic of randomization tests. Three designs are discussed: (1) Stimuli nested under treatment conditions, (2) stimuli and treatment conditions crossed, and (3) stimuli and treatment conditions counterbalanced. The treatment of stimuli as random effects in the first two designs is shown to be inadequate but is necessary in the third design.Moreover, the analysis has implications for at least two additional topics: (1) The conceptualization of the significance test as a randomization test when no randomization takes place, e.g., in quasi-experiments and correlational studies, and (2) the validity of aggregating over subjects and stimuli (referred to as aggregation validity). Aggregation validity has to be distinguished from internal validity and is shown to be one aspect of the concept formerly known as external validity, albeit without appeal to some underlying population.
The present paper starts with the following methodological question: Under which circumstances is it reasonable to consider stimuli in experimental studies to be random samples drawn from an underlying population of stimuli. By implication these stimuli would have to be treated as random factors in a mixed- or random-effects ANOVA-model. The answer to this question has to consider the following basic question: What is the exact nature of the supposed randomness of random effects? The Discussion focuses on the methodological implications and presuppositions of this assumption of randomness rather than on technical problems of the random-effects model, like the distributions of Quasi-F statistics.