Educators strive to create “assessment cultures” in which they integrate evaluation into teaching and learning and match assessment methods with best instructional practice. But how do teachers and administrators discover and negotiate the values that underlie their evaluations? Bob Broad’s 2003 volume, What We Really Value, introduced dynamic criteria mapping (DCM) as a method for eliciting locally-informed, context-sensitive criteria for writing assessments. The impact of DCM on assessment practice is beginning to emerge as more and more writing departments and programs adopt, adapt, or experiment with DCM approaches.

For the authors of Organic Writing Assessment, the DCM experience provided not only an authentic assessment of their own programs, but a nuanced language through which they can converse in the always vexing, potentially divisive realm of assessment theory and practice. Of equal interest are the adaptations these writers invented for Broad’s original process, to make DCM even more responsive to local needs and exigencies.

Organic Writing Assessment represents an important step in the evolution of writing assessment in higher education. This volume documents the second generation of an assessment model that is regarded as scrupulously consistent with current theory; it shows DCM’s flexibility, and presents an informed discussion of its limits and its potentials.

ORGANIC WRITING ASSESSMENT

UTAH STATE UNIVERSITY PRESS

Contents

1 Organic Matters: In Praise of Locally Grown Writing Assessment Bob Broad.......................................................................................................12 The Journey is the Destination: The Place of Assessment in an Activist Writing Program-Eastern Michigan University Linda Adler-Kassner and Heidi Estrem........................143 DCM as the Assessment Program: Mid Michigan Community College Barry Alford.....................................................................................................374 Designs on Assessment at UNR: University of Nevada, Reno Jane Detweiler and Maureen McBride....................................................................................525 Assessment Changes for the Long Haul: Dynamic Criteria Mapping at Indiana University Purdue University Indianapolis Susanmarie Harrington and Scott Weeden.....................756 Putting Placement on the Map: Bowling Green State University Eric Stalions.....................................................................................................1197 Bookend Bob Broad..............................................................................................................................................................154References........................................................................................................................................................................160Index.............................................................................................................................................................................164About the Authors.................................................................................................................................................................166

Chapter One

In his book Organic Matters (2001), farmer Henry Brockman criticizes the USDA definition of "organic" (grown without chemical pesticides or synthetic fertilizers) as dangerously weak. He points out that most commercially grown organic produce purchased at grocery stores lacks flavor and nutrition just as much as most commercially grown non-organic produce. Both these kinds of food are produced industrially with the goal of high yields, and with similar costs to the environment, to the flavor and nutritional value of the produce, and ultimately to consumers.

In other words, what Michael Pollan (2006) calls "industrial organic" agriculture enacts nearly (but not quite) as dramatic an abandonment of the human values and purposes of farming as does the dominant form of industrial agriculture, which also uses pesticides and petro-chemical fertilizers. Both forms of industrial farming ultimately fail to preserve or protect the land, and both fail to nourish the customer optimally; and both fail for the same reason: the pursuit of greater profits.

Brockman argues for (and practices) a tougher, more comprehensive standard of organic agriculture. This higher standard requires farmers to protect the ecosystem, select plant varieties for nutrition and taste rather than appearance and durability, and establish direct farmer-to-consumer connections through farmers' markets and Community Supported Agriculture co-operatives like those through which Brockman sells his produce. In my favorite passage from his slim book, he lays out the unanswered questions that prevented him from ever eating an organic tomato, imported from South America, brought to him by his sister one winter day a few years ago.

How could I [eat it]? I knew nothing about that tomato.... [Its] life history was a cipher to me. Who planted it? Who picked it? What kind of soil was it grown in? How was it fertilized? Irrigated? How many people had touched it on its long journey to my kitchen counter? How long had it sat in a box? Was the hangar, plane, truck, warehouse, cooler it sat in fumigated with noxious chemicals? How much fuel had been burned on its way from a field in Chile to my counter in Congerville [Illinois]? I had no idea what the answers to all these questions might be. This tomato was just too far removed from me and my life for me to eat. (Brockman, 1)

If the tomato comes from too far away for him to know its story, if the circumstances of its production and delivery to his home are, in Brockman's words (echoing Emmanuel Levinas's [1987] ethical philosophy) overwhelmingly "faceless," then Brockman would simply rather do without it altogether-even on a cold, tomato-barren December day.

Among educational leaders and reformers, the phrase "learning culture" is now commonplace (Shepard 2000). As educators, we nurture and grow our students' knowledge, skills, attitudes, and dispositions. We carefully tend the learning environment, and we provide our students with the best resources available to nourish their curiosity, understanding, and active participation in democratic citizenship. Among those of us interested particularly in assessment, we strive to create "assessment cultures" (Huot 2002) in which educators integrate their evaluations into teaching and learning (and vice versa) and match their assessment methods with best instructional practices (and vice versa).

Of course not everyone favors this "home gardening" approach to learning and evaluation. As this book goes to print, commercial testing corporations are eagerly inviting us to out-source writing (and other) assessments to their computerized systems (Ericsson and Haswell 2006). Note, for example, this recent postcard from the Educational Testing Service: "How long does it take you to evaluate an essay? Instantly ... using Criterion(tm) Online Writing Evaluation ..." (One's imagination flashes irresistibly to a hard, pale, joyless, imported tomato.) Or consider this subject line in a recent e-mail message from another evaluation corporation: "Faculty Unburdened: Assessment Made Simple in 5 Steps." Many of us feel troubled by such fast-food-style offers to make assessment faster and simpler by splitting it off from the rest of our work as educators. In such a climate, we need to recall and listen to other voices urging us to re-capture, re-coup, and harness organic, localized assessment to nourish productive teaching and learning.

In 1989, Egon G. Guba and Yvonna S. Lincoln published their book Fourth Generation Evaluation. Making good on the promise to their readers of a book "dramatically different from any other book about evaluation that you have ever read" (7), the authors issue a manifesto for a revolution in evaluation as a scholarly discipline and as an institutional practice. Though the paradigmatic and philosophical basis for-and performance of-this revolution is complex, one feature of it clearly marks it as a precursor to the evaluative approaches illustrated in the book you are now reading. Guba and Lincoln emphasize that the methods and findings of their evaluative system "are inextricably linked to the particular physical, psychological, social, and cultural contexts within which they are formed and to which they refer" (8). The authors go on to insist that much of the positive value of fourth generation evaluation comes precisely from the impossibility (their word-and their italics) of generalizing its methods and findings-which are focused on achieving a negotiated, value-pluralistic, site-specific consensus-across dissimilar contexts.

The consensus [achieved] is the product of human conceptual exchange in a particular setting; it is thus unlikely that this same consensus would necessarily help other persons make sense of their settings. (Guba and Lincoln, 8, emphasis original)

Guba and Lincoln adamantly oppose the importation of evaluative methods or findings across institutions or cultures. Their fourth generation evaluation is a militantly local, organic assessment practice.

Seven years after Guba and Lincoln's call to evaluative rebellion, Pamela Moss (1996) extended a more moderate and inclusive invitation to those in educational assessment to open our minds to

less standardized forms of assessment that honor the purposes teachers and students bring to their work ... [and] the complex and subtle ways that assessments work within the local contexts in which they are developed and used. (Moss 1996, 20)

Moss reinforces the democratizing spirit of her call for home-grown assessment by invoking the classic ethnographic imperative to "understand what the actors-from [their] own point of view-mean by their actions" (21). Moss explains how ethnographers use the term "emic" to refer to interpretations offered by participants in a particular context or culture, as distinct from the "etic" interpretations typically offered by outsiders and experts. The next generation of educational measurement, Moss insists, needs to privilege emic meanings and values.

Richard Haswell's collection Beyond Outcomes: Assessment and Instruction within a University Writing Program (2001) is also rich with calls for healthful alternatives to industrial and commercial writing assessment:

All good assessment is local (xiv)

Our moral is that writing teachers should be leery of assessment tools made by others, that they should, and can, make their own (14)

Everywhere people will prefer known brands to locally grown assessments (39)

Haswell's book urges us to steward the distinctive "climate," "ecology," and "ecosystem" (62, 67) of assessment and learning in our organizations.

One of the strongest voices promoting home-grown assessment culture can be heard in Brian Huot's Re-Articulating Writing Assessment (2002). At one point, Huot puts it this simply: "we need to use our assessments to aid the learning environment for both teachers and students" (8). In the chapter "Toward a New Theory for Writing Assessment," Huot offers his now-familiar list of five features characterizing the newly emerging paradigm in writing assessment, four of which precisely match farmer Brockman's core values: site-based, locally controlled, context-sensitive, and accessible.

The co-authors of the present volume carry forward this quest for locally produced writing assessment. In describing the dynamic criteria mapping (DCM) project faculty undertook at Mid Michigan Community College, Barry Alford gives a slightly different twist-and name-to organic assessment culture: he calls it "smart assessment."

What I find most attractive about [DCM] is that it fits my sense of "smart" assessment, assessment that makes the context, environment, or institution smarter and more reflective. (personal communication)

In similar terms, Eric Stalions speaks of the "symbiotic relationship" his participants at Bowling Green State University envisioned between placement assessment and course-based teaching and learning: each endeavors to help the other grow and thrive. Symbiotic, smart, organic, and locally grown: those are the qualities we seek in our assessments.

The unanswerable questions ("Who planted it? Who picked it? Will it satisfy and nourish the eater?") that left Henry Brockman's long-distance tomato sitting uneaten on his kitchen counter are the same questions that lead Guba and Lincoln, Moss, Haswell, Huot, Alford (2007), Stalions (this volume), and many others in the field of writing assessment to reject generic, faceless, commercialized, off-the-shelf assessments and instead to grow their assessment cultures locally and (by Brockman's rigorous definition) organically.

THIS BOOK'S ROOTS

In What We Really Value: Beyond Rubrics in Teaching and Assessing Writing (2003), I presented a critique and proposal that carried forward the relatively young tradition of local and organic assessment culture described above. I argued that, despite the significant benefits of traditional rubrics, they are too simple and too generic to effectively portray the educational values of any specific classroom, department, or program. As an alternative, I urged colleagues in composition and rhetoric to implement a process called dynamic criteria mapping (DCM).

Inspired by Guba and Lincoln's Fourth Generation Evaluation (1989) and Glaser and Strauss's grounded theory (1967), the DCM approach promotes inductive (democratic) and empirical (ethnographic) methods for generating accurate and useful accounts of what faculty and administrators value in their students' work. Educators, I claimed, have ethical, civic, and professional obligations to discover, negotiate, record, and publish the values underlying their teaching and evaluation. Finally, I argued that both the inductive and empirical characteristics of DCM made it a process superior to that by which traditional scoring rubrics are developed, and so I called DCM the necessary next step beyond rubrics in the evolution of assessment.

In the closing chapter of that book, I predicted that DCM would yield six distinct professional benefits for faculty and administrators:

Improve student learning

Provide drama, excitement, and surprise (for faculty participants)

Boost pedagogical and evaluative self-awareness

Improve teaching and assessment

Increase validity and truthfulness

Promote buy-in (especially by non-tenure-line instructors)

I ended my book with an invitation to readers to move beyond traditional assessment practices that over-simplify learning, teaching, and assessment, and to "embrace the life of things."

Early reviewers of the book were unsure, however, whether or how to accept these invitations, whether to enter the embrace. (The following analysis of the reception of What We Really Value is adapted directly from Chapter 2 of Stalions 2007.) White (2004) and Johnson (2004) were the most skeptical. White described DCM as "impressive" yet also "rather daunting" and "impractically complicated" (115). Johnson dismissed DCM as "too much work" (184) for writing program administrators.

Others saw more potential in the proposed methods. Strauch (2004) and Durst (2006) saw DCM as a new approach that would mark the end of rubrics altogether. Eliason (2004) and Beason (2005) cast DCM slightly differently, both using the term "alternative" to describe its relationship to traditional assessment methods. Kalikoff (2004), meanwhile, called DCM a "claim for alternative twenty-first century assessment."

One of the most interesting patterns of response concerned the relationship reviewers perceived between DCM and traditional rubrics. Myers (2003) saw DCM as a "new rubric," and Belanoff and Denny (2006) also described the outcome of the DCM process as another kind of rubric, albeit one "that will be applicable only within the context in which it is created" (135). The co-authors contributing to the book you are now reading struggled with exactly this concern: whether their processes and outcomes were enough unlike rubrics to qualify as dynamic criteria maps. At the conclusion of this introductory chapter, I offer my reflections on their quandary.

While the various reviewers of What We Really Value reported feeling skeptical, interested, puzzled, anxious, inspired, or blas, the co-authors of the current volume gathered the will and invested the time and energy to actually put DCM to use. They found my analyses, suggestions, and claims inviting and provocative enough that they adapted the DCM process to their distinctive purposes, needs, contingencies, and contexts. In the following chapters, they explore and discuss what they discovered and achieved when they carried out the second and third generations of dynamic criteria mapping in their college and university writing programs (see "The First Three Generations of DCM Application" in Stalions 2007). In my judgment, their discussions provide solid evidence to validate and confirm several, though perhaps not all, of my hopes and claims for DCM. You, the readers, will judge for yourselves. Meanwhile, and more important, the contributors to this volume generated exciting new insights of their own regarding homegrown, inductive assessment.

In 2002, in reviewing the manuscript of What We Really Value, Susanmarie Harrington quoted Marge Piercy's poem "To Be of Use" and predicted that the forthcoming book would prove useful to those concerned with the healthfulness of the relationship between teaching and assessing writing. In the prologue to the book published the following year (2003), I turned Susanmarie's blessing back onto the readers of the book as a benediction for the work I hoped they might do with it. As the book's author, I had little power to determine whether and how DCM would be put to use in the world. Only the contributors to this book-along with others not included here, who are putting DCM into action in yet more settings and more ways-could make Susanmarie's prediction come true. I am grateful to them all for launching their DCM projects and for studying and reporting on those projects to create this book.

At the following five institutions (presented alphabetically by co-authors' last names), this book's contributors adapted, enacted, and innovated on theories and strategies about which they had read in What We Really Value.

Local DCM leaders /co-authors of this Institutions book

Linda Adler-Kassner and Heidi Estrem Eastern Michigan University (EMU)

Barry Alford Mid Michigan Community College (MMCC)

Jane Detweiler and Maureen McBride University of Nevada, Reno (UNR)

Susanmarie Harrington and Scott Weeden Indiana University/Purdue University at Indianapolis (IUPUI)

Eric Stalions Bowling Green State University (BGSU)

Below are brief sketches of the institutional and scholarly contexts in which each co-author (or team of co-authors) conducted their DCM projects.

(Continues...)

Excerpted from ORGANIC WRITING ASSESSMENTby BOB BROAD LINDA ADLER-KASSNER BARRY ALFORD JANE DETWEILER HEIDI ESTREM SUSANMARIE HARRINGTON MAUREEN MCBRIDE ERIC STALIONS SCOTT WEEDEN Copyright © 2009 by Utah State University Press. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.