Over the last week I’ve been marking exams & the experience has led me to think (yet again) about the question of moderation. More precisely, of moderating exams – both the questions, & the marking itself. I’m beginning to think that this is a foreign concept for many teachers in undergraduate papers. (Graduate papers are a different kettle of fish – written exams are sent out so the marking can be moderated, and theses have external examiners as well as being marked in-house.)
Over the years that I’ve been teaching at the tertiary level I’ve seen some pretty awful practices: questions that are so poorly worded as to be quite ambiguous (if I’m course coordinator I take the liberty of rewriting these…); the same questions used year after year (so when papers are available in the library students catch on to this & can simply prepare & memorise answers); questions that require a single, rote-learned word or phrase to answer yet carry the same marks as a question that requires some thought and understanding to answer well… Which generates questions in response: why do people write the questions that they do? What sort of learning do these practices encourage in students? Why don’t we have some formalised system of moderating the papers prior to the exam? (You could probably add more but I don’t want this post to be toooo long!)
Now, here’s why I’m asking (& attempting to answer!) these questions – up until this year I was involved in setting assessment at a national level for our secondary school examinations, plus I’ve also taken done some work looking at Unit Standards at the tertiary level. One of the big differences between secondary and undergraduate exams is that the papers themselves are very closely moderated. Drafts are closely examined by a number of people & the examiner has to be able to justify why they’ve written a particular question in the way that they have. Ambiguities are removed, language is tightened up, examples are scrutinised for relevance and usefulness – and to be sure that they permit discrimination ie the ability to distinguish betwwen the excellent, the middle-of-the-road, and the just-getting-by students. And the questions themselves are typically supported by some contextual information, the philosophy being that at least some of the time we should be looking at students’ understanding of a topic and not simply their ability to recall facts in a sort of soundbite way. (I sometimes wonder what students who’ve experienced that system think, when they come to uni & hit a different set of assessment practices…)
I suspect that the main reason that this isn’t done for many university exams is that those setting the papers haven’t had any training in doing it. Usually someone’s been hired onto the staff on the basis of their research experience; if they’ve taught before that’s fine but the focus has only recently begun to swing to teaching. And if they do have prior teaching experience, I’d be willing bet that ‘experience’ is the key word ie they’ve picked it up as they went along. There’ll be previous tests/exams to go on for examples in setting assessment & that’s probably what new appointees base their own assessment practices on. The trouble is that this isn’t the best way to develop good assessment practices. That, plus time pressures (multi-choice & one/a few word answers are faster to mark than any sort of extended or open-ended question), leads to overuse of some of the sorts of questions I was complaining about at the start.
Which sort of leads onto an understanding of ‘curriculum’. From tearoom chats, it seems to me that for a fair number of my colleagues see ‘curriculum’ as being ‘the facts that we teach’. In fact it’s so much more. I guess one way of bringing folks to realise this would be to say, OK, what attributes do you want our graduates to have, when they finish studying. (We’ve actually got a list of these on our ‘graduate profile’.) The ensuing list will include things like practical skills, communication skills, the ability to think critically etc. So the response to that is, how are students going to pick them up? For example, they’re never going to start thinking critically about the things they’re learning until they get a clear signal that this is valued (eg via exam questions that allow them to demonstrate that skill). Developing those attributes is also part of the curriculum, and helping students to develop them is also part of our job. How we teach is also part of it: we need to take care to model the skills and attributes that we wish to see in our students.
And it’s important to be aware of that, because teaching methods and assessment practices combine to shape student learning. Which is why that ‘same questions every year’ approach is such a concern. (People can – and do – complain that the NCEA, with its limited set of Achievement Standards that focus on only some areas of the curriculum, drives an undesirable focus on learning just what’s needed to pass the exam. But having questions that change little, if at all, from year to year does exactly the same. If students know that Jim Bloggs only ever asks a particular set of questions, of course they’re likely to focus on learning only what they need to answer them! They may even write answers ahead and commit them to memory. And if the questions encourage shallow, rote learning, then all the other interesting things Jim’s said during the year will fall by the way (and indeed, you have to wonder whether he has a set of learning outcomes in mind when writing his lectures & and his tests…). Surely we want more than this from our students?
So by now you’ll have guessed that I think we do need some form of moderation for undergraduate exam papers. It doesn’t need to be external – it could simply be a brief meeting of those involved in teaching, to go over the paper and be sure that the questions are going to elicit the sort of responses that we really value. Which, of course, fits within discussions around curriculum – which need to be beyond just the indivdual papers. Which is going to get quite involved… I think I’ll just go & have a nice lie-down while I contemplate this prospect in all its glorious complexity :)