Two further testing strategies

From the hypothesis testing point of view the decomposition of the SP-lin given in (2), (3), and (4) (Testing Strategy TS1) is adequate and exhaustive, meaning that any (statistical) information needed to decide on the statistical prediction and the corresponding psychological hypothesis is contained in the two tests. But often questions arise which should and can, in addition to strict hypothesis testing, be taken into consideration when decomposing statistical predictions such as the SP-lin. Usually, these additional questions first refer to the type of trend possibly responsible for the deviations (if such occur) and second to the experimental conditions in which deviations occur. To answer these questions in a test-based manner, more than the two tests from Testing Strategy TS 1 are necessary, which means there will be a greater cumulation of the statistical error probabilities and/or . But this is the usual price you have to pay for more information. This possible disadvantage, however, can be compensated for by enlarging sample size (power analysis; see Cohen, 1988 [11]; Hager, 1992 [22]).

In the first case the global hypothesis of no deviations from the predictions should be decomposed in K-2 partial hypotheses, each concerning one trend component (maximum number of partial hypotheses: K-1), or, alternatively, in as many partial hypotheses as refer to meaningfully interpretable trend components plus a further partial hypothesis referring to all higher-order components (see, e.g., Keppel, 1973, pp. 127-128 [33], and Myers & Well, 1991, p. 216 [51], on this point).

Thus, given a particular prediction (linear in our case) and further questions concerning the type of possible deviations from linearity, the SP-lin should be decomposed into directly testable partial hypotheses, leading to Testing Strategy TS 2, which is closely related to Expression (2):
 

The tests concerning the derived null hypotheses can be performed as two-sided t or as (one-sided) F tests, in which case the trend contrasts have to be squared. If a study consists of more than three experimental conditions the tests for the strategies, TS 1 and TS 2, are based on different numerator degrees of freedom, that is, on different probabilistic testing conditions, and can entail different decisions (see, e.g., Kirk, 1982, p. 156 [36]; Maxwell & Delaney, 1990, p. 226 [45]). For this reason, these two procedures should be differentiated and are considered as different strategies.

In the case of deviations from the predictions, neither Testing Strategy TS 1 nor Testing Strategy TS 2 makes a test-based identification of the corresponding experimental conditions where these deviations arise possible. If the experimenter wants to obtain this information in addition to testing a priori hypotheses, Testing Strategy TS 3, which is more closely related to Expression (3), is preferable. According to this strategy, the test of the predicted trend component is followed by a separate statistical hypothesis for each experimental condition, postulating that there is no deviation from the prediction:
 
The rationale underlying the preceding decompositions calls for combining the tests with the strict decision rule: 'Only in case of the acceptance of and of retention of all (sufficient power provided) should strict linearity be inferred.' This results in the following decomposition of the SP-lin, leading to K+1 partial hypotheses:
 
As a consequence of this decision rule, a strictly (positive) linear trend should not be inferred if one or more of the alternatives is accepted. Although this finding contradicts the prediction, the particular tests planned allow for the identification of the experimental conditions in which the deviations from the predicted trend occur. The researcher should then try to find out possible reasons for the deviations in these experimental conditions. A possible application of this strategy is addressed by Keppel (1973, pp. 90-91) [33]. Testing Strategy TS 3 may lead to an overall decision concerning the SP-lin different from the overall decisions made using Testing Strategies TS 1 and TS 2, since the one-sample t tests rest on probabilistic testing conditions different from those of the other tests.

The testing strategies proposed in the preceding paragraphs can also be applied if two or more quantitative hypotheses which aim at the same phenomenon, but postulate different functional rules are to be tested in one experiment (see Hager, 1993 [23]). Moreover, they can be are generalized to other designs than the one chosen here (see Hager, 1992 [22]).

Further testing strategies which systematically aim at the two central features of quantitative trends can be constructed quite easily but will not be considered here, and nor will those methods of estimating parameters from the data which lead to F tests with reduced numerator degrees of freedom be dealt with (e.g. Kirk, 1982, pp. 159-161 [36]). In addition, Cohen and Cohen (1983, pp. 242-252) [12], Lee (1975, pp. 307-313) [39], Maxwell and Delaney (1990, chap. 6) [45] and Winer et al. (1991, pp. 234-236) [68] discuss various models for trend analysis, especially with respect to determining .

If a functional rule can be used to predict means, their rank order and the distances between them, it also enables the prediction of the magnitude of variances, correlations, and so on from a psychological hypothesis. Although these values, exactly predicted from theory or hypothesis, can be used as effect sizes in power analysis or sample size determination, they cannot eliminate the arbitrariness inherent in specifying effect sizes prior to experimentation, as is sometimes claimed (cf. Cohen, 1988) [11]. This arbitrariness is introduced once more when specifying the effect sizes for deviations from predictions. These effect sizes are also necessary, but cannot be predicted from psychological theory; instead, they must be chosen primarily, if not exclusively according to methodological or other considerations, such as the availability of subjects. Arbitrariness cannot be banned from statistics (see Gigerenzer, 1993; Hager, 1992) [22].

The hypotheses and the proposed decompositions of the statistical predictions have been summarized in Table 1. It should be stressed, by the way, that I only consider statistical hypotheses and tests that are necessary and adequate with respect to given psychological hypotheses formulated in advance (and with respect to certain additional questions). This focus should not prevent the researcher from additionally performing those tests which she or he thinks appropriate to gain additional information not (directly) connected to the psychological hypothesis that is to be examined (see above).
Some testing strategies for qualitative trends will be addressed next.

 
Table 1: Decomposition of a statistical prediction concerning a linear trend (SP-lin) into testable a priori hypotheses