Score Reliability: Contemporary Thinking on Reliability Issues - Softcover

Thompson, Bruce

Softcover

ISBN 10: 0761926267 ISBN 13: 9780761926269

Publisher: SAGE Publications, Inc, 2002

Synopsis

"This is a book that should be on the desk of anyone truly concerned with reliability. The whole question of conditional reliabilities is current and important; and, the question of reliability generalization is being opened out and moving away from Cronbach′s approach. The topic is an important one."

--Richard L. Gorsuch, Director of Research in Graduate School of Psychology, Fuller Theological Seminary

 

Should a high school diploma be awarded to students based on their score on a final exit exam? Should businesses deny employment to people based on their score on a personality test? In a world of "high stakes" testing, it has become more important than ever to make certain the scores on which we base our decisions are reliable. Aimed at helping researchers create and evaluate scores better, this reader presents the basic concepts of classical (or "true score") and modern ("generalizability") test theory. Beginning with a review of reliability and validity issues in measurement, the book covers score reliability, reliability induction, and reliability generalization. Exercises with sample data are included at the end of each section so readers can demonstrate knowledge of the principles.

About this title

About the Author

Bruce Thompson is (a) a former member of the Council of the American Educational Research Association (AERA), (b) a former nominee for AERA President, and (c) a former editor of AERJ:TLHD (as well as 3 other journals, including EPM for 9 years). Bruce is especially known for (a) his work on effect sizes, and (b) his contributions to creating the LibQUAL+(R) protocol, completed in ~20 language variations by more than 1,200,000 academic research library users at more than 1,100 institutions from around the world.

Excerpt. © Reprinted by permission. All rights reserved.

Preface

We engage in measurement every day. We may weigh ourselves each morning. We look at attractive people and estimate how tall they are in relation to us. Or we look at an appealing dessert and estimate how many inches eating the big splurge will add to some cherished portion of our anatomies.

Some of us are consistently accurate in estimating calories (or their consequence: inches) from visual inspection of desserts. Some of us are accurate some of the time and wildly inaccurate at other times, with no discernable pattern. Such are the questions of reliability (and unreliability).

When the consequences of somewhat unreliable measurement are relatively minor, imperfect measurement may be of little concern. But when we make "high stakes" decisions (e.g., whether to turn off the respirator because the measure of brain wave activity indicates the patient has died, or whether a high school diploma should be awarded), we had better be careful that the scores upon which we base our decisions are sufficiently reliable that our resulting decisions are reasonable.

Estimating the reliability of scores can present challenges. And as Cunningham (1986) noted, "Despite its being relatively easy to compute, there are few subjects in the field of measurement more difficult to understand than reliability" (p. 101). This reader is designed to explain basic concepts of score reliability in an accessible manner.

Specifically, the reader presents basic concepts of (a) score reliability, (b) reliability induction (i.e., inferring that scores are reliable by invoking reliability coefficients from a prior study, such as a test manual), and (c) reliability generalization (RG). RG is a meta-analytic technique in which different studies (not different people) provide the basis for exploring score reliability.

Many of the chapters in this reader first appeared in Educational and Psychological Measurement, a leading journal in measurement and statistics that was founded more than 60 years ago by a very influential psychologist named Frederic Kuder. The citations for chapters originally appearing as articles are indicated in footnotes at the start of these chapters.

The chapters originally appearing as articles have been only lightly edited. For example, citations originally appearing as "in press" have been updated. Page numbers in the original publication are indicated within narrative in bold square brackets (e.g., [157]).

To minimize excessive redundancy, some content from reprinted articles has been eliminated. These excisions are indicated with ellipses in bold square brackets (e.g., [. . .]). However, some redundancy has been maintained across chapters, so students who are reading only selected chapters may do so without omitting important concepts.

The reader has been written with the premise that you already know some basic concepts in statistics (e.g., the variance, the SD, and their symbols) and some analyses (e.g., the basic elements of an analysis of variance), although you are not expected to be an expert in these topics.

The reader covers basic concepts in "classical" (or "true score") test theory (e.g., Chapter 1). The reader also covers the basic elements of "modern" test theory (e.g., "generalizability" theory). But the reader does not cover all aspects of measurement, such as "item response theory" (or "latent trait measurement"). These IRT topics are important, but are covered elsewhere in more technical and comprehensive books (e.g., Crocker & Algina, 1986), chapters (cf. Cantrell, 1999; Hennard, 2000), or articles (Fan, 1998; MacDonald & Paunonen, in press).

The reader is not intended as a primary textbook for measurement classes. Instead, it is designed as a supplementary resource that fleshes out the key reliability concepts in an accessible manner.

The Exercises in the reader were developed to promote just such reflection. They focus on understanding coefficient alpha, and on the construction of confidence intervals for reliability coefficients. They also seek to promote personal impressions regarding the quality (or lack thereof) of the treatments of measurement issues within the published literature, and some skills for conducting "reliability generalization" (RG) studies.

Hopefully the reader will help you to think about these various issues. Thinking is what in no small part helps make for good research, and for better measurement as well!

Bruce Thompson

Edition details

• Publisher: SAGE Publications, Inc
• Publication date: 2002
• ISBN 10: 0761926267
• ISBN 13: 9780761926269
• Binding: Paperback
• Language: English
• Edition number: 1
• Number of pages: 295