Often times, people are confused by a statistical interaction pattern they find - or rather: by the lack of it. Let's say you have two potentially explaining variables X1 and X2 and a dependent variable Y. You estimate a regression model with X1, X2 and their product X1*X2 as predictors and Y as the criterion. You find that the p value associated with the interaction term is far from the 'holy hurdle' of p = .05: not significant by a long shot. So the decision is clear: I cannot claim that an interaction of the size found is far enough from zero to assertively claim an interaction effect (at least not with a type I error of .05 and a reasonable type II error = 1 - β). So, it is more reasonable to consider X1 and X2 having individual, additive effects on Y - in other words: X1 and X2 only have true main effects on Y.
Let's have an example (The R code for this example is available at https://raw.githubusercontent.com/johannjacoby/interaction_and_slopes/master/interaction%20&%20slopes.R.
These are the results:
Clearly, the results indicate that the interaction term is not significant and small, so that it could easily be explained by random fluctuations (around a true zero "interaction" in the results). This essentially means: There is not interaction of X1 and X2, the slopes of one of the predictors at different values of the respective other do not systematically differ as a function of this other predictor. But Slope of X1 @ X2 = +1SD (from now on: "high slope") is not significant with a p value = .5915 and Slope of X1 @ X2 = -1SD (from now on: "low slope") is significant with p = .0140! So it appears that the low slope and the high slope are different after all - the former is not significant, the latter is significant. So we might be tempted to ignore the non-significant interaction and simply claim that we found differential effects of X1 on Y, conditional on the value of X2. But I argue that the implicit reasoning behind this is fundamentally flawed and will elaborate on this argument.
Notice how above, we chose +1SD and -1SD as the conditional values of X2 at which we test the slope of X1. But these values of X2 are rather arbitrary. We could just as well consider the slope of X1 @ X2 = mean of X2 (from now on: "mean slope") vs. the low slope. We have the latter from above:
and we obtain the mean slope from the original basic regression model that we used to test the interaction in the first place:
yielding the result:
So, comparing the mean slope vs. the low slope, we see that they are both significant. So, according to the mere comparison of significance decisions, they are "the same" - the simple slopes do not differ. One might argue that one of the p values is smaller than the other, but p values and their differences are a very poor indicator of an actual difference as they are not a linear function of the actual effect size. The difference between p = .30 and p = .40 is not the same as the difference between p = .11 and p = .01, so gauging whether two effects differ by eyeballing two p values and making a guess as to whether they actually are "very different", "not so much, but still different", or "not really different at all" will not cut it as a reproducible and transparent decision rule for the scientific test of a theory. In addition, p values are by definition highly dependent on sample size. So a difference of the same magnitude between two slopes (i.e., a difference in bs) might look like a huge difference with N = 300 (because the p values are far apart), but with N = 120, the difference in p values might not look so big anymore. This is of course true for any p value from a statistical test, but the problem is exacerbated if you look at differences between effects.
So, if we compare the low and high slopes by a rather haphazard guess about the difference between p values (absolute diff(p) = .577 in the case of comparing the low and high slopes), we get the result "the slopes differ"; but if we compare the mean slope and the high slope, we get the result "they don't differ, they are both significant". One might argue that this is unfair, because the difference in X2 between the mean and the high slopes is much smaller than the difference in X2 between the low and the high slopes. That is unquestionably true, but who is to say which difference in X2 is the "right one"? Also, we could chose another set of values of X2 on which we condition the slope of X1 and obtain a similar result that the slopes are not really different (according to simple p value comparison). We can estimate the simple slopes at any values of X2 we wish to, so we could pick X2 = -2SD of X2 and X2 = mean of X2. These two conditional values are exactly as far apart as + and - 1SD above (i.e., 2SDs), so the comparison of these two simple slopes is just as "fair" toward a difference between the slopes as that of the high and low slopes above:
The slope of X1 @ X2 = -2SD (i.e., the "-2 slope") and the slope of X1 @ X2 = mean of X2 (i.e., the "mean slope") are both significant:
And their p values only differ by a miniscule .0013. That surely is not a difference in simple slopes that should be taken serious.
Thus, even if we compare two simple slopes that are as far apart in X2 as the low and high slopes, now we have to come to the conclusion (if we use the crude p value comparison) that the slopes are not different, and both not significant. The comparison between the p values again is not a big help: this time the difference between p values of the slopes is diff(p) = .0013, while in the comparison between low and high slopes it was .577. How is one to determine whether these differences in p are the same or different? That's right, it will become a mess to continue with comparing p values by approximate visual inspection and making transparent scientific decisions based on these p value decisions.
Things are, however, different if we look at the difference in b, diff(b) in the two cases. In the comparison of the low and high slopes above, we obtained a difference of diff(b) = -0.96 and in the comparison -2 slope vs. mean slope we obtained the exact same difference: -0.96. It appears more sensible to use the difference in b as a representation of the difference in slopes - at least the difference in slopes thusly indicated is the same when the difference in the conditional values of X2 is the same. But how can we test if this difference in b is substantial, or allows us - based on a clear and transparent criterion - to say: "the two slopes at two different values of the moderator X2 are different"? After all we have no p value for the difference in b! Or do we? Yes, we do! Well, not a p value for the comparison between the low and high slopes or that comparison between the -2 and mean slopes, but we have one for any pair of simple slopes that are 1 unit of X2 apart: The p value for the interaction term in the very first basic regression analysis. The interaction term tests whether the difference of two simple slopes of X1 on Y that are estimated or calculated at two different conditional values of X2 that are exactly one unit of X2 apart is significant. So this does not compare the low and high simple slopes, but since the low and high simple slopes are exactly 2SD of X2 apart and SD is a stable multiple of the unit of X2, the test is good enough. Remember that after all the interaction term is linear - this means that the increase in the simple slope of X1 as X2 becomes larger (or the decrease as X2 becomes smaller, or even - if the interaction is negative: the increase in the simple slope of X1 as X2 becomes smaller and the decrease in the simple slope of X1 as X2 becomes larger) is the same no matter which two conditional values of X2 one might chose, as long as they are equally far apart, i.e., 1 unit.
In sum: the comparison of two p values in order to assess whether simple slopes are different is not to be recommended as it gives different comparison values for slopes of the same distance in X2 and p values are not very easy to compare. In contrast, the p value of the original interaction allows to make a decision for or against the hypothesis of "differing slopes" that will be the same for any pair of conditional X2 values that are the same distance apart. This p value of the interaction is a focused and concise basis for such a decision and does not require an additional decision about whether two p values are close, far apart or basically the same. The interaction term and its statistical properties is what the comparison of simple slopes should be based on, not the isolated slopes or pairs of them. If the interaction term is not significant - by the logic of significance testing - the slopes are not different. If the interaction term is significant, then any pair of slopes whose conditional values of the moderator are one unit apart, are different - no matter what their individual p values are.
There are two more general points that can be taken away from this:
- Comparing things
- If you want to compare two means, test their difference against zero (instead of testing each individually against zero and then visually inspecting a difference between the two results). For group mean comparison, this test of the difference is achieved by an independent sample t-test.
- The same goes for simple slopes: Test their difference against zero (instead testing each individually against zero and then visually inspecting a difference between the two results). This test of the difference between slopes is achieved by the statistical test of the interaction effect.
- And it goes on: if you want to know whether two two-way interactions differ, do not conduct two two-way analyses and then visually compare their differences - instead, look at the three-way interaction - it tells you whether the two-way interactions differ.
- If you want to compare to structural equation models, do not estimate each of them and then visually inspect the individual RMSEA values (or any other fit indices) - instead, test the difference in fit between the models, by testing the difference in χ².
- If you want to compare two correlations between X and Y within different groups of Z, do not compute each correlation within groups of Z and then visually inspect differences between the two correlations - rather, enter X, Z, and their product interaction term as predictors into a regression model predicting Y, then the interaction will tell you whether the association between X and Y differs depending on Z.
- If you conduct a mediation analysis based on recommendation formulated most prominently in Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173-1182., you might be tempted to engage in comparing the c and c' paths (the total effect and the direct effect remaining after statistically controling M in the prediction of Y, respectively). It is still widespread to draw substantial conclusions from the comparison of these two paths regarding "complete" or "partial" mediation. In this logic, mediation is "complete" if the total effect is significant, but the direct effect is not. Imagine a case where c is associated with a p value of p=.04 (significant) and c' has a p value of p=.06 (not significant). According to the "complete"/"partial" mediation logic such a difference would have to qualify as "complete". On the other hand, a case where the total effect has p=.06 (not significant) and a positive sign, and the direct effect has a negative sign and also p=.06 (so is also not significant but by far in the opposite direction), even though the indirect effect is much, much larger than in the former example, one would have to conclude "no indirect effect/no mediation" here. And finally, the total effect may be very very large with a miniscule p value, and the direct effect could be also significant, with a larger p value (but still smaller than .05), with a huge difference between these two effects, but you would have to conclude "only" "partial" mediation. One reason for this mess is that the distinction between "partial" and "complete" mediation relies on comparing individual p values instead of simply basing decisions on a test of the difference. This difference, in normal cases with continuous variables, is exactly the indirect effect, i.e., a ×b and can be tested in a simple elegant fashion without the visual inspection of p values that very poorly represent effect sizes and differences between effects. More on this in Hayes, A. F. (2009). Beyond Baron and Kenny: Statistical mediation analysis in the new millennium. Communication Monographs, 76, 408-420. and Rucker, D. D., Preacher, K. J., Tormala, Z. L., & Petty, R. E. (2011). Mediation analysis in social psychology: Current practices and new recommendations. Social and Personality Psychology Compass, 5(6), 359-371.
- Other combination of interaction and slopes significance decisions
In general, comparing two effects (or any other statistics) by just laying them side by side and approximately assessing a difference in p values is never a good idea. What counts is whether the difference between the two is significant, not the individual significance decisions for the two statistics to be compared. This general principle is very clear to anyone if we look at the comparison of two means. Suppose you want compare two groups regarding a dependent variable that ranges from -16 to +16. So you do a t-test within each group and test whether the mean of the dependent variable differs from zero:
You get the following results for the two groups:
So the group means are different, right? The group mean in Group 1 is not significantly different from zero, but the mean in Group 2 is nice and significant. And the graph confirms this - one of the means is essentially zero, the other on is significantly different from zero:
Right? Of course not. Nobody in their right mind would accept such a comparison of two group means by looking at the difference in p values. The correct way to compare two means is not looking at the difference of p values associated with the test of individual group means against zero, but testing the difference between the means:
This gives us a test of whether the difference between the means is different from zero:
So, even though the group means appear to be different, based on their individual p values, they are not different: the difference between them is not significantly different from zero, and in the logic of significance testing the means should be considered not different. The best you can do to characterize the means is to take the same estimate for both: the grand mean. Any deviations from that grand mean can plausibly be attributed to chance.
Of course nobody would proceed as described here (i.e., visually comparing p values for individual group means and drawing conclusions regarding the difference between them). But to the same degree that this strategy appears nonsensical for two group means, it also should appear nonsensical for the comparison of two slopes, two interactions, two structural equation models or the comparison of a total and direct effect in mediation analysis:
Of course, it goes the other way around: you might find yourself in a situation where the X1×X2 interaction term is significant, but the simple slopes you compute seem to not differ - they are both significant, e.g. one has a p value of p = .03 and the other one of p = .001. This case is just the other side of the coin of the issue discussed above. Of course the simple slopes can differ between two conditional values of X2 that are one unit apart, and they can both be significant, but one of them is larger than the other. Remember that the p value does not give you a good idea of how large an effect is, it essentially tells you whether you should be weary that an effect of a given size that you found might be plausibly explained as a random deviation from an actual null effect. So again, if you want to know the difference between effects, you need to look at their difference and test that difference rather than 'looking' at whether the difference between p values gives you a tingly gut feeling. And if the interaction is significant, two individually very clearly significant simple slopes at X2 = mean of X2 and X2 = mean of X2 + 1 will also be statistically different. They are both so far from zero that you may deem it implausible for any of them to come from a true null distribution, but one of them just may be larger than the other.
And finally you may come across cases where none of the simple slopes you elect to compute may be significant, but the interaction is significant. This may of course be due to your choosing to examine slopes at conditional values of X2 that are quite close together, but also in cases where the conditional values of X2 (for the simple slopes of X1) are reasonably far apart. In an example:
In this particular data set, the following estimates and test results are obtained:
The interaction is significant, so the slopes differ. The simple slopes at X2 = -1SD and X2 = +1SD however are both not significant. But look at their signs: they are opposite. So even though the simple slopes may not be strictly significantly different from zero, they differ from each other very clearly. After all the difference in b is much larger than either simple slope b coefficient by itself. The interaction term picks up this difference and is clearly significant.
Posts
