Talking Teaching

November 9, 2010

moderation in all things

Over the last week I’ve been marking exams & the experience has led me to think (yet again) about the question of moderation. More precisely, of moderating exams – both the questions, & the marking itself. I’m beginning to think that this is a foreign concept for many teachers in undergraduate papers. (Graduate papers are a different kettle of fish – written exams are sent out so the marking can be moderated, and theses have external examiners as well as being marked in-house.)

Over the years that I’ve been teaching at the tertiary level I’ve seen some pretty awful practices: questions that are so poorly worded as to be quite ambiguous (if I’m course coordinator I take the liberty of rewriting these…); the same questions used year after year (so when papers are available in the library students catch on to this & can simply prepare & memorise answers); questions that require a single, rote-learned word or phrase to answer yet carry the same marks as a question that requires some thought and understanding to answer well… Which generates questions in response: why do people write the questions that they do? What sort of learning do these practices encourage in students? Why don’t we have some formalised system of moderating the papers prior to the exam? (You could probably add more but I don’t want this post to be toooo long!)

Now, here’s why I’m asking (& attempting to answer!) these questions – up until this year I was involved in setting assessment at a national level for our secondary school examinations, plus I’ve also taken done some work looking at Unit Standards at the tertiary level. One of the big differences between secondary and undergraduate exams is that the papers themselves are very closely moderated. Drafts are closely examined by a number of people & the examiner has to be able to justify why they’ve written a particular question in the way that they have. Ambiguities are removed, language is tightened up, examples are scrutinised for relevance and usefulness – and to be sure that they permit discrimination ie  the ability to distinguish betwwen the excellent, the middle-of-the-road, and the just-getting-by students. And the questions themselves are typically supported by some contextual information, the philosophy being that at least some of the time we should be looking at students’ understanding of a topic and not simply their ability to recall facts in a sort of soundbite way. (I sometimes wonder what students who’ve experienced that system think, when they come to uni & hit a different set of assessment practices…)

I suspect that the main reason that this isn’t done for many university exams is that those setting the papers haven’t had any training in doing it. Usually someone’s been hired onto the staff on the basis of their research experience; if they’ve taught before that’s fine but the focus has only recently begun to swing to teaching. And if they do have prior teaching experience, I’d be willing bet that ‘experience’ is the key word ie they’ve picked it up as they went along. There’ll be previous tests/exams to go on for examples in setting assessment & that’s probably what new appointees base their own assessment practices on. The trouble is that this isn’t the best way to develop good assessment practices. That, plus time pressures (multi-choice & one/a few word answers are faster to mark than any sort of extended or open-ended question), leads to overuse of some of the sorts of questions I was complaining about at the start.

Which sort of leads onto an understanding of ‘curriculum’. From tearoom chats, it seems to me that for a fair number of my colleagues see ‘curriculum’ as being ‘the facts that we teach’. In fact it’s so much more. I guess one way of bringing folks to realise this would be to say, OK, what attributes do you want our graduates to have, when they finish studying. (We’ve actually got a list of these on our ‘graduate profile’.) The ensuing list will include things like practical skills, communication skills, the ability to think critically etc. So the response to that is, how are students going to pick them up? For example, they’re never going to start thinking critically about the things they’re learning until they get a clear signal that this is valued (eg via exam questions that allow them to demonstrate that skill). Developing those attributes is also part of the curriculum, and helping students to develop them is also part of our job. How we teach is also part of it: we need to take care to model the skills and attributes that we wish to see in our students.

And it’s important to be aware of that, because teaching methods and assessment practices combine to shape student learning. Which is why that ‘same questions every year’ approach is such a concern. (People can – and do – complain that the NCEA, with its limited set of Achievement Standards that focus on only some areas of the curriculum, drives an undesirable focus on learning just what’s needed to pass the exam. But having questions that change little, if at all, from year to year does exactly the same. If students know that Jim Bloggs only ever asks a particular set of questions, of course they’re likely to focus on learning only what they need to answer them! They may even write answers ahead and commit them to memory. And if the questions encourage shallow, rote learning, then all the other interesting things Jim’s said during the year will fall by the way (and indeed, you have to wonder whether he has a set of learning outcomes in mind when writing his lectures & and his tests…). Surely we want more than this from our students?

So by now you’ll have guessed that I think we do need some form of moderation for undergraduate exam papers. It doesn’t need to be external – it could simply be a brief meeting of those involved in teaching, to go over the paper and be sure that the questions are going to elicit the sort of responses that we really value. Which, of course, fits within discussions around curriculum – which need to be beyond just the indivdual papers. Which is going to get quite involved… I think I’ll just go & have a nice lie-down while I contemplate this prospect in all its glorious complexity :)

Advertisements

6 Comments »

  1. I completely agree with the need for moderation and training in exams. I did recently have a conversation with a colleague about the reuse of questions that made me think about them a bit differently. He said he had a professor in college that had certain questions everyone knew he asked every single time. The professor had decided what he thought were the absolutely critical points from his classes and wrote a set of questions to cover them. Naturally, these questions were not multiple choice or a simple fill in the blank type, but they were fairly simple list and explain type questions. The result was that even years later, his students could pop out the answers to those questions in their sleep. They may not have remembered anything else of the class, but they knew those points cold.

    Certainly repeating a test verbatim from year to year is a bad idea. But this discussion made me think that there may be a place for repeating certain questions from year to year, if those questions are the key things you want them to remember above all else.

    Comment by jdmimic — November 10, 2010 @ 8:26 am

    • if those questions are the key things you want them to remember above all else
      I think that’s the key, isn’t it? But it does require careful thought about what you’re teaching, & why, in order to identify what those things are (& also why you think they’re important). Which is where learning outcomes come into the picture. I could go on about this stuff for ages but it would probably put everyone to sleep :)

      Comment by alison — November 10, 2010 @ 8:33 pm

  2. I have question about question types, since we are talking about moderating tests. My school has banned any multiple choice question that is designed as “All of the following are true EXCEPT…”. All questions must have four demonstrably wrong answers and one clearly true answer, so we can’t have them tell us what is wrong with a problem, only what is right. We also can’t have them distinguish between a simply correct answer and the best answer. In my opinion, this is terrible. We are training doctors and we are teaching them to think in only this one specific way. Moreover, we are teaching them there is always only one correct answer and that everything that is technically correct is of equal importance. For instance, we can’t ask a question such as, “what is a likely route for blood to get from the toe to the heart?” The best answer of course is through the inferior vena cava. But since one could say correctly that blood plasma could pass through the capillaries into the lymph system or any of innumerable hypothetical routes, the question does not have one clearly correct answer and so can’t be asked.

    I would like to know what people here think of questions like this. Are questions asking students to pick out the incorrect answer that bad? Anyone know what the literature says on this?

    As a side note, this topic came up because I had a few questions in my lecture designed this way and I was told they were not acceptable forms of questions for the students. I found this rather interesting as I pulled the questions verbatim from their textbook.

    Comment by jdmimic — November 11, 2010 @ 10:57 am

    • I haven’t come across that exclusion before. Personally I don’t use multichoice questions as I find it difficult to set really good ones :) Colleagues do, though – and quite often they’ll use questions where there’s more than one correct answer. They tend to use different phrasing though, along the lines of ‘which of the following is/are correct?’. This still means that students need to be able to identify what’s wrong.

      I agree 100% that teaching students to think in one particular way is not good – what we need to cultivate is flexibility. Otherwise they’re going to run into trouble in unfamiliar situations and new contexts, where the ability to think around a problem and recognise alternatives is very helpful. Similarly the idea that there’s always only one correct answer…

      Interesting indeed that your ‘bad’ questions came from the textbook!

      Having said that I don’t use MC questions – I’ve just started using the ‘PeerWise’ system (http://peerwise.cs.auckland.ac.nz) with a couple of groups that I work with. It was developed by Paul Denny, who’s a comp sci lecturer at the University of Auckland, but as you’ll see from its home page it’s gone global :) Teachers set it up but it’s intended for students, who write MC questions about whatever they’re working on for their peers to answer and comment on. When Paul showed it to us the question at the top of everyone’s minds was, ‘but what if the answers the question-writer provides are wrong???’ His response was that he’d looked into that one (as had someone at Queensland Uni, quite independently) & that about 10% of questions had incorrect ‘right’ answers. And that in every case he looked at this was corrected by other students :)So I think this is a really useful peer-teaching/support tool.

      Comment by alison — November 12, 2010 @ 9:56 am

  3. Thanks for sharing that link. I will have to look into it.

    I should also probably mention that we are allowed to ONLY ask multiple choice. All test questions must be multiple choice with one correct answer and four false ones. I was told this is the style of questions on the national medical board exams, so all our questions must reflect their board exams, no exception. Talk about teaching to the test.

    Comment by jdmimic — November 12, 2010 @ 2:56 pm

    • Yikes! I remember back when I was a first-year student & the end-of-year biology paper was all multiple choice. Fairly easy to do well in, I thought at the time, if you did a bit of study ahead & looked at the questions from the last few years’ previous exams. It didn’t exactly encourage a lot of reading around the subject :(
      Teaching to the test, indeed. It’s rather depressing to hear that, actually. As you said earlier, it encourages a style of thinking that is surely undesirable.

      Comment by alison — November 12, 2010 @ 7:55 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: