<Order DETECT or Books> <Home Page> <Info. Page> <Contact us: Info@levelsofanalysis.com> <Return to list of comparisons>

Comparison of WABA and DETECT to James' "rwg" approach

Description of the James' "rwg" approach

James et al (1984) developed a measure of agreement among raters in a particular group called "rwg". Each group gets an "rwg" value. The custom has been to try and find a value above .7 in most groups. The notion is that if there is that much agreement in each group then, it makes sense to aggregate the data.

Compatibility of within and between analysis with James' "rwg"

WABA does not view all within-group variance as error (see the section on the ANOVA). So in cases where there is substantial between-group variation or covariation, then the use of "rwg" provides one way to see how much agreement occurs within each group. In cases where there is mainly variation within groups, "rwg" will indicate a lack of relationship but WABA will view this as a within-group level effect.

Schriesheim, Cogliser, and Neider (1995) examined the relationship between "rwg" and the various WABA indicators. James (1998) reacted to these findings. Then, Schriesheim et al. (1998) responded to James and concluded that both approaches provide different information. Of particular interest is the conclusion by Schrieshim et al (1998). "... when conflict in "rwg" and WABA findings are obtained, the WABA results will be more useful overall." This simply means that WABA provides additional and important information that is not available from "rwg". Cohen, Doveh, & Uri Eick (2001, p.298), point out that, "However, as pointed out by James (1998) rwg and WABA seldom are competing systems."

Misunderstandings and Misconceptions about WABA/DETECT

Statement #1 :Use "rwg" instead of WABA.

Answer: If you find significant between-group variation, then "rwg" can be used in addition to WABA. For example if you find "wholes" based on WABA I, then "rwg" would provide an indicator of agreement in each group. You could use "rwg" in addition to WABA. If you find parts, "rwg" is not an appropriate indicator for parts. The same is true for the equivocal case. Thus, the response to statement #1 depends on whether the data allow the use of "rwg." In addition if you specify a theory based on "rwg" WABA provides some additional information but it does not directly examine "rwg". This means that WABA and "rwg" are different and have different purposes--not that one is better than the other. See Schriesheim et al. (1995, 1998) for additional information.

Statement #2: George and James (1993) point out that range restriction across group scores may lead to erroneous WABA I conclusions (since this leads to conclusions that groups are not present).

Answer: If your hypothesis is that there are valid differences between groups, it is necessary to try and assure that the variance occurs where you assert that it will occur. For example, if you are interested in group differences on some variable, this requires sampling groups that you believe will differ on that dimension. In some cases that may be difficult. If you can not obtain some variation on the variable, then the effects will not be adequately represented in the data. Stated simply, WABA will not compensate for poor sampling relative to your hypotheses.

Statement #3. George and James (1993) point out that WABA when it identifies parts does not indicate the degree to which units vary (i.e., whether there is an interaction.)

Answer: Shriesheim et al. (1995) illustrate one way to test for this type of interaction or homogeneity within groups. The additional tests are helpful, if one is concerned about whether all groups show similar results within groups.

References

Cohen, A. Doveh, E., & Eick, U. (2001). Statistical properties of the rwg index of agreement. Psychological Methods, 6, 297-310.

George, J. & James, L. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 789-804.

James, L. (1998). Implications of a multiple-levels-of-analysis Ohio state leadership study for estimating interrater agreement. In F. Dansereau and F. Yammarino (Eds.) Leadership: The multiple level approaches (Part A). Stamford CT: JAI Press (pp.61-64)

James, L., Demaree, R., & Wolf, G. (1984). Estimating within-group interater reliability with and without response bias. Journal of Applied Psychology, 78, 306-309.

Schriesheim, C., Cogliser,C., and Neider, L. (1995). Is it "trustworthy." A multiple levels of analysis reexamination of an Ohio State leadership study. Leadership Quarterly, 6, 111-145.

Schriesheim, C., Cogliser,C., and Neider, L. (1998). "Trustworthy" is a judgment call!In F. Dansereau and F. Yammarino (Eds.) Leadership: The multiple level approaches (Part A). Stamford CT: JAI Press (pp.65-72).

Yammarino, F. & Markham, S. (1992). On the application of within and between analysis: Are absence and affect really group based phenomena? Journal of Applied Psychology, 77, 168-176.

<Order DETECT or Books> <Home Page> <Info. Page> <Contact us: Info@levelsofanalysis.com> <Return to list of comparisons>