<Order DETECT or Books> <Home Page> <Info. Page> <Contact us: Info@levelsofanalysis.com> <Return to list of comparisons>
Comparison of WABA and DETECT to Bliese and Halverston's (1998) Modified (ANOVA) approach
Description of the Bliese approach
Bliese and Halverston (1998) argue against the use of eta squared as used in WABA in favor of the intraclass correlation (ICC(1)).In addition they argue for the use of ICC(2) for an estimate of reliability of means. Finally, Bliese (2000) argues for the use of an adjusted between-group correlation presumably instead of the between-group correlation used in WABA.
One basis for the replacement of eta squared by the intraclass correlation is a simulation that shows as the size of the group increases eta-squared becomes closer to the intraclass correlation. This is actually a well known consequence of the equations for eta-squared and the intraclass correlation (see Snijders & Bosker, 1999, pp.30-31). The point is somewhat moot because the statistical significance of the intraclass correlation and eta squared is the same F-ratio. Thus, Bliese is against the use of Cohen's f value to test for the strength of effects. A particularly salient point about the intraclass correlation was made by Dansereau, Alutto, & Yammarino (1984) as well as Snijders and Bosker (1999) who show that the intraclass correlation can be expressed in terms of the F-ratio. The latter statisticians (p.22) state, "These formulae show that a high value for the F-statistic will lead to large estimates for the between-group variance as well as the intraclass correlation , but that group sizes, as expressed by cell size, moderate the relation between the test statistic and the parameter estimates." Thus, the intraclass correlation is not a good indicator for indexing the magnitude of effect size independent of sample size since it varies by it. Dansereau and Yammarino (2000) have also described the reasons to retain Cohen's approach to test for practical significance.
The second part of the Bliese (2000) approach suggests the use of ICC(2) to adjust obtained between group correlations. Snijders and Bosker (1999, p.26) show that if the number of individuals per group is large that ICC(2) approximates 1.0. They provide a graph showing that as the number of cases per group increases, reliabilities as low as 0.1 or 0.4 approximate 1.0. This suggests that the Bliese's approach will take very small variations between groups as indicating a between effect.
The previous statement is in line with the proposed adjustment to the between group correlation (Bliese, 2000, p. 373). Here an obtained correlation is divided by the square root of the product of the multiplication of the ICC(2) values for the two variables of interest (x and y). For example, if the ICC (2) for variable x is 0.2 and the ICC (2) for variable y is 0.2 and the obtained between-group correlation is 0.21, the adjusted between-group correlation would be 1.05. If the obtained between-group correlation equaled 0.4 the resulting adjusted between-group correlation would be 2.0. Sources that describe the characteristics of these correlations that go beyond values of 1 were not found.
Compatibility of Within and Between Analysis (WABA/DETECT) with the Bliese approach
Bliese's approach essentially takes the indicators from WABA and adjusts them so that they are more compatible with inferring a wholes effect. This is done by assuming that any variation within groups is error. (Indeed, the Bliese and Halverson (1998) simulation assumed that the means were valid and scores were created that varied randomly within-groups. This assumes that within-group variance can only be error.) Thus, the adjustments remove parts. In addition ICC (2) which is known to approximate one as group sizes increase is used as a way to show an increasing number of stronger between-group effects when group sizes increase. Finally, between-group correlations as low as 0.2 are adjusted so that they also can viewed as indicating a correlation between variables of one. The new indicators do not enter into the partitioning of correlations and address a question other than that addressed by WABA.
Misconceptions and misunderstandings about WABA/DETECT
For a more detailed discussion about the Bliese approach and WABA/DETECT see Dansereau and Yammarino (2000)
Statement #1
Bliese (2000, p. 362) states " In short researchers are probably best advised to simply avoid using and interpreting eta-squared values , particularly because ICC (1) values are easily estimated."
Response: ICC (1) is based on the assumptions of the analysis of variance that within-group variance is error. If you have a wholes effect and a reviewer insists that you include ICC(1) values, you can argue with an obtained wholes effect using WABA that there is a reason to assume that within-group variance is error and thus you used an intraclass correlation. If you have an equivocal or parts effect, you may find that the intraclass correlation will show a stronger effect than the between-eta correlation. Your inference would have to change. You can still present ICC(1). But point out the differences between the two indicators and point out the basis for assuming within-group variance is error is or is not based on the data.. Another part of the inference process in WABA is to examine the differences between the correlations of variables based on within-groups and between-groups scores. If you have stronger between group correlations, you might use that to justify the assumption of error within-groups. Alternatively, you can point out that WABA does not use a pure ANOVA framework. So, the ICC(1) does not apply. Problems are unlikely to happen when your WABA results indicate wholes.
Statement #2
Doesn't the between-eta ignore ICC (1) and ICC (2)?
Response: Snijders and Bosker (1999, p. 30) indicate that eta squared is defined as the ratio of the intraclass coefficient to the reliability of the group mean. As they state, "For large group sizes the reliability approaches unity, so the correlation ratio [Between eta squared] approaches the intraclass correlation."
Statement #3
House (1987, p.462) stated, "Consequently one has to have faith in the metric of the E-ratios (f from Cohen (1988) as descriptive statistics. But, this is possible only in the context of particular variables and error structures."
Response: There is not only one way to examine the variance between groups. Likewise, there is not only one way to decide whether there is a between-groups effect. The values of the between-group eta correlations are important because they enter into the total correlation. They are not the only ones but they are important. The value that is needed to infer an effect will always be somewhat arbitrary much like the use of .05 and .01 in statistics as House suggests. The consequences of using different cutting points for the E-ratio such as those implied by Bliese's work need to be investigated.
References
Bliese, P (2000). Within-group agreement, non-independence, and reliability. In K. Klein & S. Kozlowski (Eds.) Multi-level theory, research, and methods in organizations. San Francisco: CA: Jossey-Bass (pp.349-381).
Bliese, P. & Halverston, R. (1998). Group size and measures of group-level properties: An examination of eta-squared and ICC values. Journal of Management, 24, 157-172.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
Dansereau, F. & Yammarino, F. (2000). Within and Between Analysis. In K. Klein & S. Kozlowski (Eds.) Multi-level theory, research, and methods in organizations. San Francisco: CA: Jossey-Bass (pp.425-466).
Dansereau, F., Alutto, J., and Yammarino, F. (1984) Theory Testing in Organizational Behavior. Englewood Cliffs NJ: Prentice Hall
House (1987). Review of Theory testing in organizational behavior. Administrative Science Quarterly, 32, 459-464.
Snijders, T. & Bosker, R. (1999). Multilevel analysis. London: Sage Publications.
<Order DETECT or Books> <Home Page> <Info. Page> <Contact us: Info@levelsofanalysis.com> <Return to list of comparisons>