This is the story of the curious path that took me from a graduate school obsession with brewing and judging beer, to an interest, and eventually career, in assessment. This apparently random linkage was something I realized somewhat belatedly, well into my assessment career. But the more I started to think about it, the more complex the linkages became. Moreover, I’ve come to believe that there are some things that the world of beer judging can actually teach us about assessment too.
But let's start at the beginning. What came first was actually beer making, a hobby I acquired at the same time I started graduate school. I think it was the blending of science and art in brewing that I found most compelling, and fairly quickly I started to enter homebrew competitions. Surprisingly, there is often a shortage of beer judges at competitions, and so at my first competition I was pressed into service.
Beer judging, for the uninitiated, is not some bacchanal, but rather a highly structured endeavor. Beers are presented to the judges without any clues as to origin, at the same temperature (which varies according to style), and judges do not drink more than several ounces of each entry. In competition, beers are scored on a 50 point weighted scale covering aspects such as color, clarity, aroma, and different components of the flavor profile such as bitterness and mouthfeel. In other words, a rubric.
Another aspect of beer judging is that beers are judged based on how true they are to a particular style of beer. It is not a question of merely taste. Instead, homebrewers attempt to brew beer to match the characteristics and flavor profiles of styles like India Pale Ale (IPA), Bohemian Pilsner, Munich Dunkel, or Stout. And they are judged accordingly - a very good stout that was entered as a pilsner would earn a terrible score. In other words, standards.
Training to become a certified beer judge is more demanding than you might think. The process is governed by the national Beer Judge Certification Program (BJCP). Judges must acquire a deep understanding of all aspects of the beer making process, the different characteristics of the styles (the latest style guide includes over 100 distinct styles), and the chemical and biological origins of beer flavors both desirable and undesirable.
After months or years of preparation, aspirants take a three-hour written exam testing their knowledge of styles and brewing process, which also includes scoring several beers, including at least one that is spoiled in some way. Their score indicates what rank they can attain, but attaining most ranks also requires gaining experience points through judging at regional competitions.
And I found the complexities of judging beer to be fascinating. Judges don’t just have taste preferences, but also are sensitive to different elements in the flavor profile. I have always been particularly sensitive to staling compounds in beer that is not fresh, while my friends would be better at distinguishing different hop varieties by taste or aroma, or were more sensitive to other types of off flavors.
So, how did all these lead me to assessment? And what lessons from beer judging did I carry over into my assessment work?
Well, beer judging is not always fun - flaws in the brewing process can lead to some very off putting flavors, and judges have to train themselves to recognize those and provide feedback to the brewer. I’ve had more than a few undrinkable entries over the years. Yet the crucial point is an educational one - helping brewers get better - so the connection to teaching was immediate and obvious to me. And of course, I was introduced to the power of rubrics in that teaching capacity.
Beer judging also got me to think a lot about interrater reliability. On the one hand, rubrics and training provided a way for greater reliability, as the BJCP exam showed me. But also, it soon became obvious that *perfect* reliability across evaluators was never going to be attainable, and that was alright. Human expertise and judgement remains critical in areas where the key elements are not measurable by a machine, and that's as true for higher education as it is for beer. The outcomes we really care about in higher education - the ability to communicate, to think critically, to welcome diversity - require trained judgement: Some “noise” in the resulting data will always be present.
Another lesson was in the humanistic dimensions of standards. In the mid-90s, I became involved in the first major revision of the style guidelines where I had to convince my fellow committee members that beer styles both evolve over time, and are human constructs, not absolutes. As most of the others on the committee were engineers (which homebrewing tends to draw in inordinate numbers) who like black and white answers, this was something of an uphill battle. And despite imperfections, standards could provide crucial guides to practice, while leaving many paths open as to how to achieve them. Many different combinations of hops, malt, yeast, water and process can produce an award-winning American Pale Ale.
And that is a lesson I have thought about quite a bit in the last five years, as I have been deeply immersed in developing institutional learning outcomes for my institution. For one thing, our college competencies represent the educational expression of our institutional values. They have to be specific to our context and meaningful for our faculty and staff - and that can only emerge from a process of dialogue and communal collaboration, in which historical precedents and faculty preferences play an important role. At the same time, teaching to standards does not mean teaching exactly the same way. Instead, there are many paths a teacher can choose from in helping students to meet general standards. Vive la difference!
I don’t recommend mixing assessing student work and beer! Still, I’ve found that beer judging deepened my appreciation of fine ales and lagers at the local brewpub, and eventually led me down a professional path that I didn’t expect to take, but that has proven to be intellectually stimulating nonetheless.
