Handbook of Inter-Rater Reliability (2nd Edition) - Softcover

Kilem L Gwet

Softcover

ISBN 10: 0970806221 ISBN 13: 9780970806222

Publisher: Advanced Analytics, LLC, 2010

Synopsis

A successful data collection should always aim to uphold the integrity of a research project. In many scientific fields of study, data collection activities require adherence to guidelines and compliance with regulations. The consistency with which various observers or raters follow guidelines, and implement methods is essential for ensuring the interpretability and usefulness of data. Independent observers must be able to replicate processes and obtain comparable results. This book discusses many proven techniques and their use in evaluating the extent of agreement among observers.

The author explains in a clear and intuitive fashion the motivations and assumptions underlying each technique discussed in the book. He demonstrates the benefits of using basic level statistical thinking in the design and analysis of inter-rater reliability experiments. Interpretation and limitations of various techniques are extensively discussed.

From optimizing the design of the inter-rater reliability study to validating the computed agreement coefficients, the author's step-by-step approach is practical, easy to understand and will put all practitioners on the path to achieving their data quality objectives.

Contents

Acknowledgments .x

Chapter

1. Introduction .1
1.1 Overview2
1.2 Response Category .3
1.3 Different Reliability Types .5
1.4 Statistical Inference .7
1.5 Book’s Structure .8

2. Kappa Coefficient: A Review .11
2.1 Overview .12
2.2 Kappa for 2 Raters and 2 categories .13
2.3 Kappa for 2 Raters and Multiple Categories .20
2.4 Kappa for Multiple Raters and Multiple Categories .25
2.5 Kappa Coefficient & the Paradoxes .30
2.6 Weighting of the Kappa Coefficient .34
2.7 Some Alternative Coefficients .37
2.8 Concluding Remarks .41

3. Kappa for Ordinal & Interval Data .43
3.1 Overview .44
3.2 Generalizing Kappa for 2 Raters & 2 Categories .45
3.3 Generalizing Kappa, Pi, & BP to Interval Data & 2 Raters .48
3.4 Generalizing Kappa, Pi, & BP to Interval Data & Multiple Raters .50
3.5 Generalized Weighted Agreement Coefficients .53 3.6 Concluding Remarks .57

4. The AC1 Coefficients .59
4.1 Overview ..60
4.2 Gwet’s AC1 & Aickin’s Statistics .61
4.3 Aickin’s Theory ..65
4.4 Gwet’s Theory .68
4.5 Estimating AC1 for 3 Raters or More .73
4.6 AC2 Coefficient for Ordinal Data .76
4.7 Weighting the AC1 Coefficient .80
4.8 Concluding Remarks .82

5. Agreement Coefficients & Statistical Inference .85
5.1 The problem .86
5.2 Finite Population Inference in Inter-Rater Reliability Analysis .89
5.3 Conditional Inference .93
5.4 Unconditional Inference .104
5.5 Concluding Remarks .109

6. Benchmarking Inter-Rater Reliability Coefficients .111
6.1 Overview .112
6.2 Benchmarking the Agreement Coefficient .114
6.3 Proposed Benchmarking Method .120
6.4 Critical Value Calculation .126
6.5 Concluding Remarks .129

7. Inter-Rater Reliability: Conditional Analysis .139
7.1 Overview .140
7.2 Conditional Agreement Between 2 Raters in ACM Reliability Studies .142
7.3 Conditional Agreement Between 2 Raters in RCM Reliability Studies .147
7.4 Concluding Remarks .154

8. Variance Estimation of Conditional Agreement Coefficients .157
8.1 Overview .158
8.2 Variance Estimation .159
8.3 Variance of Reliability Coefficients in ACM Studies .161
8.4 Variance of Validity Coefficients in ACM Studies .167
8.5 Variance Estimation in RCM Studies 172
8.6 Limited Literature Review .180
8.7 Concluding Remarks .181

Appendix A: Data Tables .183
Bibliography .189
Author index .193
Subject index .195

About this title

About the Author

KILEM L GWET has a doctorate degree in Statistics from Carleton University. He is a statistical Consultant, a researcher and an instructor. He has over 15 years of experience in various industries, with clients including Booz Allen & Hamilton, American Health Information Management Association (AHIMA), Centers for Medicare & Medicaid Services (CMS), IMPAQ International and many others. He has several publications on Inter-rater and Intra-rater reliability assessment in peer-reviewed journals. He currently lives in Maryland.

From the Back Cover

A successful data collection should always aim to uphold the integrity of a research project. In many scientific fields of study, data collection activities require adherence to guidelines and compliance with regulations. The consistency with which various observers or raters follow guidelines, and implement methods is essential for ensuring the interpretability and usefulness of data. Independent observers must be able to replicate processes and obtain comparable results. This book discusses many proven techniques and their use in evaluating the extent of agreement among observers. The author explains in a clear and intuitive fashion the motivations and assumptions underlying each technique discussed in the book. He demonstrates the benefits of using basic level statistical thinking in the design and analysis of inter-rater reliability experiments. Interpretation and limitations of various techniques are extensively discussed.
From optimizing the design of the inter-rater reliability study to validating the computed agreement coefficients, the author's step-by-step approach is practical, easy to understand and will put all practitioners on the path to achieving their data quality objectives.

From the Inside Flap

This text is intended to general practitioners, researchers, students with general analytical background. Being able to read basic mathematical expressions will ease the reading without it being a prerequisite for accessing the material. The key concepts and main approaches are explained in plain language independently of the mathematical formulas. The book is full of numerical examples to show how the different techniques are implemented in practice. To facilitate the use of the techniques presented in this book, I developed a user-friendly point-and-click Excel VBA program called AgreeStat, which can be downloaded from the website agreestat.com. This program can handle a large number of response categories. It can calculate various agreement coefficients available in the literature for 2 raters or more, along with their standard errors. Conditional analysis on specific categories has been implemented as well.

Edition details

• Publisher: Advanced Analytics, LLC
• Publication date: 2010
• Language: English
• ISBN 10: 0970806221
• ISBN 13: 9780970806222
• Binding: Paperback
• Edition number: 2
• Number of pages: 208