Generative AI is challenging us to rethink a fundamental question in higher education: Are our assignments truly measuring student learning, or are they measuring something AI can now do for them? In this month’s NEean blog, board members reflect on this timely question and what it means for the future of assessment. From rethinking traditional assignments to exploring new ways to make student learning visible, their perspectives invite us to move beyond simply detecting AI use and toward designing assessments that deepen learning, critical thinking, and authentic demonstration of skills. 

Carina

AI challenges us to re-consider our learning outcomes for specific assignments to ensure that what we are asking students to demonstrate are outcomes that go beyond what can be easily produced by AI without including any evidence of the learning process on the part of the student.  Asking students to document and justify their reasoning or choices in creative production (even if the final product is AI-assisted) gives us authentic evidence of the process of inquiry and creation. We also need to bring students into the conversation about the learning outcomes to reinforce the value they bring to AI-assisted work products. 

Maura

AI challenges teaching and learning to move beyond the knowledge acquisition paradigm.  Learning should be assessed on what students are able to do in teaching environments that mirror the real world as much as possible.  Assessments at the assignment level should aim to ensure students are identifying appropriate and up-to-date information, able to think critically about other and/or missing information and frameworks that are also needed for any given inquiry, drawing accurate conclusions as they make meaning of easy-to-acquire information, making appropriate judgments and decisions, testing assumptions and determinations, and to apply appropriately to case studies and scenarios.  Recall the concept of ‘saturation’ as it applies to a review of the literature in a given field that a disciplinary expert achieves by years of study before starting to write a dissertation or before generating other new knowledge.  In short, AI jumpstarts a student’s knowledge in a given content area, essentially putting the student ‘on par’ with the disciplinary expert who has consumed all the related literature.  What does the dissertation writer do at that point?  Makes an argument that stems from her or his inquiry, gathers information and data, tests assumptions and biases, evaluates and analyzes results, identifies limitations, and thinks critically about implications for policy and practice.  AI challenges us to develop assignments that assess the accurate processing and judicious application of easy-to-acquire information to real world case studies, in other words, the implications for policy and practice. 

 

Denny

The question about AI and assignments doesn’t necessarily make me rethink how to do assessment. Indeed, for me, assessment has always been more about gaining a glimpse into the how a student is developing their thinking processes rather than their content mastery. To put it another way, the higher levels of Bloom’s taxonomy have always motivated my approaches to assessing student work and development. That is, I’ve always viewed the best assessment practices to be ones that reveal the students’ progress on how and why they think rather than what they think. The goal is to reveal their individual (and often unique)  knowledge production, not simply their mastery of content. To be sure, they need the content to demonstrate their proficiency in the analysis and synthesis stages of critical reasoning, but the best assessment protocols and the best assignments compel students to not simply regurgitate the content. Instead, they are compelled to employ that content, applying it in expected and unexpected ways, oftentimes creating new, authentic knowledge. So, if we are creating assignments that can be completed by agentic and/or generative AI, then we are only assessing content mastery, and as argued above, that only scratches the surface of what we really should be striving to assess. 

Jen 

We have a toolbox of wide-ranging direct and indirect measures that we can use to assess student learning and provide us with quantitative and qualitative data to understand the extent to which students have met the learning outcomes. Yet in many ways, AI throws a proverbial wrench into assessing learning. Students now have more tools that they can use to elude assessment, and when they feel stressed or overwhelmed, they might feel more compelled to take the fastest path and prompt AI to complete their work. From what I have read in higher education news reports, heard anecdotally from colleagues, and done myself, many of us have been using in-class blue book exams as a workaround. I have learned that blue book exams serve as a temporary fix, and they limit students’ time to process assessment questions, put together their responses, and revise their answers before handing in their work. We can use AI software as an opportunity to reconsider our assessment measures, make sure they align with our learning outcomes, and authentically assess learning. We can also add AI into our toolbox of assessment measures to teach content knowledge and reinforce the importance of durable skills, such as communication, information literacy, and critical thinking. AI presents us with an opportunity to leverage emerging technologies, double down on our efforts in teaching durable skills, and provide students with the skills to navigate their next pursuits.

Conversants:

Dennis (Denny) Frey, NEean's Director of Communication & Professor of History, Lasell University

Kim Hamilton Bobrow, Professor of English & Program Chair of General Studies, Connecticut State - Manchester 

Rachel Jasiczek, Professor of English & Program Chair of Writing Across the Disciplines, Connecticut State - Norwalk

In this short podcast (15 minutes or so), Kim, Rachel, and Denny chat about Peggy Maki's legacy in the world of assessment, particularly her work on assessing learning as opposed to assessing for reporting purposes.

Click here to listen.

Guests:  Kim Hamilton Bobrow, Professor of English Connecticut State-Manchester

Rachel Jasiczek, Program Chair of Writing Across the Disciplines Connecticut State

In this brief, 30-minute podcast, Denny engages Kim and Rachel in a topic very near and dear to all their hearts:  Reading in the Age of AI.  Serving two, intertwined purposes, this shortcast offers both a thought-provoking conversation about what it means to read and how we read in and across disciplines, as well as offering an example of the kinds of dialogues in the disciplines that NEean fosters at its annual spring event.

This is the story of the curious path that took me from a graduate school obsession with brewing and judging beer, to an interest, and eventually career, in assessment. This apparently random linkage was something I realized somewhat belatedly, well into my assessment career.  But the more I started to think about it, the more complex the linkages became.  Moreover, I’ve come to believe that there are some things that the world of beer judging can actually teach us about assessment too. 

But let's start at the beginning. What came first was actually beer making, a hobby I acquired at the same time I started graduate school. I think it was the blending of science and art in brewing that I found most compelling, and fairly quickly I started to enter homebrew competitions. Surprisingly, there is often a shortage of beer judges at competitions, and so at my first competition I was pressed into service.  

Beer judging, for the uninitiated, is not some bacchanal, but rather a highly structured endeavor.  Beers are presented to the judges without any clues as to origin, at the same temperature (which varies according to style), and judges do not drink more than several ounces of each entry. In competition, beers are scored on a 50 point weighted scale covering aspects such as color, clarity, aroma, and different components of the flavor profile such as bitterness and mouthfeel. In other words, a rubric.

Another aspect of beer judging is that beers are judged based on how true they are to a particular style of beer. It is not a question of merely taste. Instead, homebrewers attempt to brew beer to match the characteristics and flavor profiles of styles like India Pale Ale (IPA), Bohemian Pilsner, Munich Dunkel, or Stout.  And they are judged accordingly - a very good stout that was entered as a pilsner would earn a terrible score. In other words, standards. 

Training to become a certified beer judge is more demanding than you might think. The process is governed by the national Beer Judge Certification Program (BJCP). Judges must acquire a deep understanding of all aspects of the beer making process, the different characteristics of the styles (the latest style guide includes over 100 distinct styles), and the chemical and biological origins of beer flavors both desirable and undesirable.

After months or years of preparation, aspirants take a three-hour written exam testing their knowledge of styles and brewing process, which also includes scoring several beers, including at least one that is spoiled in some way. Their score indicates what rank they can attain, but attaining most ranks also requires gaining experience points through judging at regional competitions.  

And I found the complexities of judging beer to be fascinating. Judges don’t just have taste preferences, but also are sensitive to different elements in the flavor profile. I have always been particularly sensitive to staling compounds in beer that is not fresh, while my friends would be better at distinguishing different hop varieties by taste or aroma, or were more sensitive to other types of off flavors.

So, how did all these lead me to assessment?  And what lessons from beer judging did I carry over into my assessment work?  

Well, beer judging is not always fun - flaws in the brewing process can lead to some very off putting flavors, and judges have to train themselves to recognize those and provide feedback to the brewer. I’ve had more than a few undrinkable entries over the years. Yet the crucial point is an educational one - helping brewers get better - so the connection to teaching was immediate and obvious to me. And of course, I was introduced to the power of rubrics in that teaching capacity. 

Beer judging also got me to think a lot about interrater reliability. On the one hand, rubrics and training provided a way for greater reliability, as the BJCP exam showed me. But also, it soon became obvious that *perfect* reliability across evaluators was never going to be attainable, and that was alright. Human expertise and judgement remains critical in areas where the key elements are not measurable by a machine, and that's as true for higher education as it is for beer. The outcomes we really care about in higher education - the ability to communicate, to think critically, to welcome diversity - require trained judgement: Some “noise” in the resulting data will always be present.  

Another lesson was in the humanistic dimensions of standards. In the mid-90s, I became involved in the first major revision of the style guidelines where I had to convince my fellow committee members that beer styles both evolve over time, and are human constructs, not absolutes. As most of the others on the committee were engineers (which homebrewing tends to draw in inordinate numbers) who like black and white answers, this was something of an uphill battle. And despite imperfections, standards could provide crucial guides to practice, while leaving many paths open as to how to achieve them.  Many different combinations of hops, malt, yeast, water and process can produce an award-winning American Pale Ale.  

And that is a lesson I have thought about quite a bit in the last five years, as I have been deeply immersed in developing institutional learning outcomes for my institution. For one thing, our college competencies represent the educational expression of our institutional values. They have to be specific to our context and meaningful for our faculty and staff - and that can only emerge from a process of dialogue and communal collaboration, in which historical precedents and faculty preferences play an important role.  At the same time, teaching to standards does not mean teaching exactly the same way. Instead, there are many paths a teacher can choose from in helping students to meet general standards. Vive la difference! 

I don’t recommend mixing assessing student work and beer! Still, I’ve found that beer judging deepened my appreciation of fine ales and lagers at the local brewpub, and eventually led me down a professional path that I didn’t expect to take, but that has proven to be intellectually stimulating nonetheless.