Testing strategies for monotonic and other qualitative trends

The psychological hypothesis to be examined may state that 'the amount of retrieval (dependent variable) increases with increasing values of imagery (the independent variable).' To test this hypothesis J > 2 levels of imagery and an observable dependent variable such as 'number of words correctly remembered' are chosen. Omitting the psychological prediction referring to the observable dependent variable and the design chosen, the statistical prediction is derived from the hypothesis in an adequate and exhaustive manner. This prediction refers to statistical concepts exclusively, and since the discussion is restricted to (population) means the resulting statistical prediction states a strictly monotonic trend among the J means. This prediction has been called SP-mon in (9). Since this statistical prediction cannot be tested appropriately and exhaustively by a single test, it is then decomposed into testable partial hypotheses about focussed pair contrasts. These partial hypotheses can be tested in a way that enables unambiguous (as far as test results are concerned) and test-based decisions concerning the statistical prediction and that avoids any inconsistencies stemming mainly from data-based inferences. 'To avoid inconsistencies' simply means: Ranks are only called 'different,' if there are 'significant differences' among the means according to the usual statistical criteria and tests applied, whereby 'usual tests' refers to any two-sample test, whether it is a t test or a multiple comparison procedure on pair contrasts, each with only one degree of freedom. Such rankings are test-based. The SP-mon has already been presented above, but is given here again:

This formulation of the statistical prediction suggests a decomposition into Q = J - 1 partial hypotheses for adjacent means, conjunctively combined:

 
The Q = J - 1 directional partial hypotheses can be tested by one-sided t tests. If all Q tests come out according to predictions then a strict monotonic trend (without rank equalities and without rank inversions) can be inferred, at least within the limits of statistical error: The predicted ranks can unequivocally be assigned to the (empirical) means, symbolizing a 'significant' distance for each pair of means without, however, knowing the sizes of the distances. But this particular information is not necessary in respect to the psychological hypothesis, although it should be computed from the data at hand. Under the strict decision rule applied it is not necessary to consider all possible pair contrasts: If at least one partial prediction, each referring to one testing instance, does not show up, the SP-mon1 should be rejected, and this decision cannot be modified by showing that there are significant differences between non-adjacent means.

The more experimental conditions that have been chosen, the greater the cumulation of error probabilities, but also the more severe the test of the psychological hypothesis, all other things being equal. The cumulation can be compensated for by an adequate adjustment, for example, by the Dunn-Bonferroni method or an improved version of it (see, e.g., Kirk, 1982, pp. 106-111 [36]; Westermann & Hager, 1986 [66]; Winer et al., 1991, pp. 158-166 [68]).

If the SH-mon in (8) is connected with a lenient decision rule the acceptance of at least one partial alternative out of J-1 partial hypotheses () suffices to accept the respective SP-mon2 (). The decomposition is the same as before, but the decision rule is different. This means that the SP-mon2 is more easily accepted than the SP-mon1, but the test of the respective psychological hypothesis is less strict than with the SP-mon1. An even more lenient decision rule gives leave to 'look for' the one necessary conforming result among all possible pairs of means which leads to the SP-mon3. If tested according to this prediction the psychological hypothesis has an even less severe test to survive than if tested by the SP-mon2.

In the derivation and testing of the SP-mon2 or the SP-mon3 the problem of possible rank inversions has not been discussed. There are basically two options concerning rank inversions. First, they are accepted if they occur when testing the SP-mon2 or the SP-mon3. Second, they are or at least a maximum number of them is exluded a priori by a corresponding extension of the decision rule. In this instance additional tests should be planned referring to these inversions. Let us return to the SP-mon2 and extend its decision rule to handle possible rank inversions; this extension leads to the SP-mon4, which deliberately allows for a maximum of rank inversions a priori, suggesting the following decomposition:
 
The total number of tests to be considered and planned is . The retention of one or more of the implies that either '' is true (indicating equality of ranks), or that '' holds, indicating a rank inversion. In order to enable a test-based differentiation of the two possibilities, the tests on the corresponding partial null hypotheses should be performed for these pairs of means. If more than of these tests come out significant, the SP-mon4 should be rejected. This testing strategy is in accordance to the proposals made by Shaffer (1972, 1974) [59].

The decision rule applied in the SP-mon4 is more lenient than the one in the SP-mon1, but stricter than the one in the SP-mon3. Because of different numbers of pairs it is difficult to say whether the decision rule of the SP-mon4 is stricter than the rule of the SP-mon2, but since the SP-mon2 allows for J-2 rank inversions at most, the SP-mon4 will most probably lead to a more severe test of the psychological hypothesis. Further decision rules or criteria concerning the maximum number of rank inversions and/or the number of pairs of means to be considered can be additionally defined, but will not discussed here. The recommendations are summarized in Table 2.

Prediction	Decomposition	Number of pairs (tests)	Kind of decision rule
SP-mon1	µj < µj' for all j,j'; j=j'-1	J-1 (J-1)	strictest possible; no rank equalities or inversions accepted
SP-mon2	µj < µj' for at least one pair j,j'; j=j'-1	J-1 (J-1)	more lenient, rank inversions permitted
SP-mon3	µj < µj' for at least one pair j,j'; j < j'	J(J+1)/2 [J(J+1)/2]	more lenient, rank inversions permitted
SP-mon4a	µj < µj' for at least one pair j,j'; j < j' and further hypotheses to exclude rank inversions (see text)	J(J+1)/2 (see text)	lenient, maximum number of rank inversions tolerated
Notes.aVarious similar kinds of predictions differ with respect to the number of pairs or testing instances considered. See text for further details.