This design results, if stimuli and treatment levels can be combined
freely, but each stimulus can be presented only once. In this design
stimuli and treatment conditions are balanced by building two sets of
stimuli and subjects (in case of two treatment levels) and crossing them.
The treatment effect equals the interaction of the dummy-coded subject (A)
and stimulus (B) sets (cf. Kenny & Smith, 1980). With respect to the two
dummy variables subjects and stimuli are nested. Equation (3) shows the
linear model for a single score 
.
In the counterbalanced design both, subjects as well as stimuli, are
randomly assigned (in the present case to two groups), that is, stimuli
and subjects are treated symmetrically. Consequently, the number of the
subjects and the stimuli should be equal. If stimuli are treated as random
effects the treatment effect entails the variance component
, just the
same way as the treatment of subjects as random effects leads to the
inclusion of the variance component 
(see Table 4). In both cases this
variance component reflects an estimation of the amount of variance that
can be attributed to the randomization of subjects and stimuli, respectively.
This reflects the ANOVA approximation of a between subjects randomization
test. As a consequence, from the perspective of the subjects as well as from
that of the stimuli, an ANOVA in which both, subjects and stimuli, are
treated as random effects is a valid conceptual approximation of a
randomization test in which the distribution of the test statistic is
calculated by permutating both stimuli and subjects. Therefore, inasmuch
as tests of significance - in the interpretation of being randomization
tests - raise the internal validity of an experiment, this is also true
for the treatment of stimuli as random effects in the counterbalanced design.
Source | df | E(MS) |
| A | 1 |  |
| B | 1 |  |
| St/B | 2(r/2-1) |
 |
| Su/A | 2(n/2-1) |  |
 | 1 |  |
 | 2(r/2-1) |  |
 | 2(n/2-1) |  |
 | 4(n/2-1)(k/2-1) |  |
and 
can not be estimated
independently with only one observation per stimulus-subject combination.
One of the disadvantages of the random-effects model has not yet been discussed (because the application of the random-effects model did not prove to be advisable): The underlying variance components model is not as robust against violations of its statistical assumptions as the fixed-effects model, which is analyzable in the context of the General Linear Model. For instance, the random-effects model is less robust against violations of the normality assumption, since the central limit theorem is not applicable to variance components, as opposed to means (e.g., Scheffé, 1963). Moreover, the distributions of the Quasi-F statistics, the basis for tests of significance, cannot be derived analytically and have to be analyzed by means of Monte-Carlo studies (e.g., Maxwell & Bray, 1986; Santa, Miller, & Shaw, 1979). For example, in the present case, the resulting Quasi-F ratio is given by Equation (4), cf. Kenny and Smith (1980).
One straightforward way to avoid such statistical problems is to confound stimuli and the subjects factor. Under these circumstances, the fixed-effects model can be applied. In the present case, however, in order to confound both factors it is not necessary to draw a new random sample of stimuli for each subject (e.g., Keppel, 1976; Coleman, 1979; Richter & Seay, 1987). Instead, it is sufficient to randomly assign stimuli to conditions individually for each subject. That is, the distribution of the test statistic is not generated post hoc by means of permutations, like in the randomization test, but the permutation is actually performed individually for each subject.