Talking Teaching

December 13, 2013

can we help students too much?

One of the (many) good things about writing exams at a national level was that it really taught me how to write a good question: one that lacks ambiguity, clearly identifies what’s being asked of the students, and gives them opportunities to demonstrate their knowledge and understanding. So I was a bit saddened when marking exams recently to find that one question, in particular, wasn’t eliciting what I’d hoped for.

You see, I’d written a question in two parts. Students were first asked to summarise their knowledge the process of gene expression, and then – using the phrase “in your answer you should also” – asked to explain how various events might introduce diversity into the genome. And for some reason many students went straight past the first part and answered the second (and generally did this very well, I might add).

It’s the first time I’ve seen this to any marked degree and it really bothered me, & I’ve talked about it at some length with a colleague. We’re beginning to wonder if maybe we give students too much information around the exams. Now that may sound wrong – you’d think that demystifying exams, in the sense of giving access to previous papers so students get an idea of what to expect, would be helpful. It also highlights key themes in the subject that we return to again & again. We don’t just leave it at this, mind you: we spend time in tuts discussing things like how to ‘unpackage’ questions and how to plan the answer before actually writing it. But we’re wondering if making previous papers readily available in the study guide is leading to students ‘picking questions’ – deciding what’s likely to be asked, on the basis of what’s been asked in the past, and then not just studying around that but perhaps writing and learning ‘model’ answers off by heart. And regurgitating that answer regardless of what the question actually requires of them.

And that sort of learning is not the sort of learning I want to encourage.

So we decided to do things a little differently with next year’s study guide & include just a single year’s exam paper in it. Students will still be able to access many more through the library’s database; they’ll just have to do a bit more work themselves. And thinking more on this as I’m writing, perhaps we should also devote a bit of tut time to explicitly recognising those themes & working with the students to build up a ‘big picture’ view of the main ideas associated with them, by way of encouraging that breadth of understanding – and emphasising that this approach should allow them to approach with confidence any question we might ask.

Your thoughts?

December 12, 2013

Evaluating teaching the hard-nosed numbers way

[This is a copy of a post on my blog PhysicsStop, sci.waikato.ac.nz/physicsstop, 10 December 2013]

Recently there’s been a bit of discussion in our Faculty on how to get a reliable evaluation of people’s teaching. The traditional approach is with the appraisal. At the end of each paper the students get to answer various questions on the teacher’s performance on a five-point Likert Scale (i.e. ‘Always’, ‘Usually’, ‘Sometimes’, ‘Seldom’, ‘Never’.)  For example: “The teacher made it clear what they expected of me.” The response ‘Always’ is given a score of 1, ‘Usually’ is given 2, down to ‘Never’ which is given a score of 5. An averaged response of the questions across students gives some measure of teaching success – ranging in theory from 1.0 (perfect) through to 5.0 (which we really, really don’t want to see happening).

We’ve also got a general question – “Overall, this teacher was effective”. This is also given a score on the same scale.

A question that’s been raised is: Does the “Overall, this teacher was effective” score correlate well with the average of the others?

I’ve been teaching for several years now, and have a whole heap of data to draw from. So, I’ve been analyzing it (for 2008 onwards), and, in the interests of transparency, I’m happy for people to see it.  For myself, the question of “does a single ‘overall’ question get a similar mark to the averaged response of the other questions?” is a clear yes. The graph below shows the two scores plotted against each other, for different papers that I have taught. For some papers I’ve had a perfect score – 1.0 by every student for every question. For a couple scores have been dismall (above 2 on average):

Capture1.JPG

What does this mean? That’s a good question. Maybe it’s simply that a single question is as good as a multitude of questions if all we are going to do is to take the average of something. More interesting is to look at each question in turn. The questions start with “the teacher…” and then carry on as in the chart below, which shows the responses I’ve had averaged over papers and years.
Capture2.JPG
Remember, low scores are good. And what does this tell me? Probably not much that I don’t already know. For example, anecdotally at any rate, the question “The teacher gave me helpful feedback” is a question for which many lecturers get their poorest scores (highest numbers). This may well be because students don’t realize they are getting feedback. I have colleagues who, when they give oral feedback, will prefix what they say with “I am now giving you feedback on how you have done” so that it’s recognized for what it is.
So, another question. How much have I improved in recent years? Surely I am a better teacher now than what I was in 2008. I really believe that I am. So my scores should be heading towards 1.  Well, um, maybe not. Here they are. There are two lines – the blue line is the response to the question ‘Overall, this teacher was effective’, averaged over all the papers I took in a given year; the red line is the average of the other questions, averaged over all the papers. The red line closely tracks the blue – this shows the same effect as seen on the first graph. The two correlate well.
Capture3.JPG
So what’s happening. I did something well around 2010 but since then it’s gone backwards (with a bit of a gain this year – though not all of this year’s data has been returned to me yet). There are a couple of comments to make. In 2010 I started on a Post Graduate Certificate of Tertiary Teaching. I put a lot of effort into this. There were a couple of major tasks that I did that were targeted at implementing and assessing a teaching intervention to improve student performance. I finished the PGCert in 2011. That seems to have helped with my scores, in 2010 at least. A quick peruse of my CV, however, will tell you that this came at the expense of research outputs. Not a lot of research was going on in my office or lab during that time.  And what happened in 2012? I had a period of study leave (hooray for research outputs!) followed immediately by a period of parental leave. Unfortunately, I had the same amount of teaching to do and that got squashed into the rest of the year. Same amount of material, less time to do it, poorer student opinions. It seems a logical explanation anyway.
Does all this say anything about whether I am an effective teacher? Can one use a single number to describe it? These are questions that are being considered. Does my data help anyone to answer these questions? You decide.

December 9, 2013

shaking up the academy? or how the academy could shake up teaching

Last week I spent a couple of days down in Wellington, attending the annual symposium for the Ako Aotearoa Academy. The Academy’s made up of the winners of the national Tertiary Teaching Excellence Awards, so there are around 150 or so of us now. While only 35 members were able to make it to this year’s event (& the executive committee will survey everyone to see if there’s a better time – having said that, everyone seems so busy that there’s probably no date that would suit everyone!), we had a great line-up of speakers & everyone left feeling inspired & energised. I’ll blog about several of those presentations, but thought I would start with one by Peter Coolbear, who’s the director of our parent body, Ako Aotearoa.

Peter began by pointing out that the Academy is potentially very influential – after all, it’s made up of tertiary teachers recognised at the national level for the quality of their teaching, & who foster excellence in learning & teaching at their own institutions.  But he argued – & I agree with him – that there is room for us to become involved in the wider scene. Peter had a number of suggestions for us to consider.

First up, there’s a lot going on in the area of policy – are there areas where the Academy might be expected to have & express an opinion? For example

  • There’s the latest draft of the Tertiary Education Strategy (TES), which “sets out the Government’s long-term strategic direction for tertiary education; and its current and medium-term priorities for tertiary education.” There’s a link to the Minister’s speech announcing the launch of the draft strategy here.
  • In addition, the State Services Commission’s document Better Public Services: results for New Zealanders sets out 10 targets across 5 areas. Targets 5 & 6 are relevant here as they are a reference point for government officials looking at evidence for success in the education sector. (Such scrutiny is likely to become more intense in light of the 2012 PISA results, which have just been made public.) Target 5 expects that we’ll “[increase] the proportion of 18-year-olds with NCEA level 2 or equivalent qualification”; #6 is looking for an increase in ” the proportion of 25 to 34-year-olds with advanced trade qualifications, diplomas and degrees (at level 4 or above)”. This will increase the pressure on institutions to increase retention & completion rates – might this have an effect on standards?
  • There’s also the requirement to achieve parity of success for ‘priority’ learners, especially Maori & Pasifika – this is priority #3 in the TES. (Kelly Pender, from Bay of Plenty Polytech, gave an inspirational presentation on how he weaves kaupapa Maori into pretty much everything he does in his classroom, in an earlier session.) And it’s an important one for us to consider. Peter cited data from the Ministry of Education’s website, ‘Education Counts’, which showed significantly lower completion rates for Maori & Pasifika students in their first degrees compared to European students, and commented that this will likely become a major issue for the universities in the near future.
  • If we’re to meet those achievement requirements, then how institutions scaffold learners into higher-level study, through foundation & transition programs, will become increasingly important. What are the best ways to achieve this?
  • Peter predicted increased accountability for the university sector (including governance reform). Cycle 5 of NZ’s Academic Audits has begun, and “is to be framed around academic activities related to teaching and learning and student support.” This is definitely one I’d expect Academy members to have an opinion on!
  • He also expects strengthened quality assurance processes throughout the education sector: this suggests a stronger (& more consistent) role for the  NZ Qualifications Authority, with the development of partnership dialogues across the sector (ie including universities).

Then, at the level of the providers (ie the educational institutions themselves – & that’s not just the polytechs & universities), we have:

  • a targeted review of qualifications offered at pre-degree level – there’s background information here;
  • a government-led drive to get more learners into the ‘STEM’ subjects (science, technology, engineering, & maths) – this poses some interesting challenges as, at university level, we’re seeing quite a few students who’ve not taken the right mix of subjects, at the right NCEA level, to go directly into some of the STEM papers they need for, say, an engineering degree;
  • the rise in Massive Open On-line Courses, or MOOCs. (I find these quite strange creatures as they are free to the student and typically attract very large enrolments, but also apparently have very low completion rates. What’s in them for the institution? A good way of offering ‘taster’ courses that hook students in?)
  • the likelihood that we will see the development of a system for professional accreditation of tertiary teachers (I’ve written about this previously and will write another post fairly soon, as accreditation was the subject of a thought-provoking session at the symposium);
  • how we achieve protection of academic standards – it’s possible that government policies (eg those linking funding to completion & retention rates) may result in a tendency to exclude of underprepared kids &/or lowering standards – neither is desirable but both are possible results of those policies.

That’s a big list and the Academy can’t do everything! So, what should it focus on? (This is not a rhetorical question – it would be great to get some discussion going.) The Academy, in the person of its members, is effectively a resource; a body of expertise – can it become a ‘go-to’ body for advice? Speaking personally I think we need to make that shift; otherwise we remain invisible outside our individual institutions & the teaching-focused activities we’re involved in, & in a politicised world that’s not a comfortable thing to be. Can we, for example, better promote the significance of teaching excellence outside the education sector? Become involved in the discussions around & development of any accreditation scheme? Develop position papers around maintaining teaching excellence in the context of the new TES?

What do you think? And what shall we, collectively, do about it?

December 5, 2013

nz’s pisa rankings slip, & the soul-searching begins

Filed under: education, science teaching — Tags: , , , , — alison @ 11:06 am

The latest PISA results are out, and NZ – despite remaining in the ‘above the average’ group for OECD countries – has nonetheless  slipped in this measure of achievement in reading, maths administered by the Programme for International Student Assessment . This is of concern, & there are probably multiple complex causes for our decline. Certainly the previous PISA commentary (2009) recommended that we pay attention to matters of inequality (There’s interesting commentary here, & also on the RNZ website.)

This morning’s Dominion-Post (I’m in Wellington at the moment, at a teaching symposium) carries a story giving a primary-teaching perspective.There are two key issues here: many primary teachers lack a science or maths background; and primary teachers in general are not well supported to teach these specialist sujects. (The removal of specialist science advisors - something I’ve commented on previously – did not help things.) This is important, because if students don’t gain a good understanding of these subjects – and good experiences of them! – during primary school, then they’ll basically be playing catch-up when they arrive in specialist secondary school classrooms.  Sir Peter Gluckman’s suggestion (in his report Looking ahead: science education in the 21st century) that each primary school have a ‘science champion’ would help here, but in the medium-to-long term it would probably be even better if intending primary school teachers received much greater exposure to the STEM subjects to begin with.

Should we worry? Yes, but I definitely agree with Fiona Ell, from the University of Auckland, who’s quoted in this morning’s Herald as saying:

People get very hung up on the ranking … because it’s like a Top of the Pops top 10 thing. I don’t think they should be ignored … but knee-jerk reactions to rankings are really dangerous in education systems.

So, there are issues that we need to address, and as Fiona’s pointed out, there are no quick fixes – we need to deal with them in a considered way that includes as many variables as possible (i.e. not just practices in schools).

One of those issues is highlighted by Sir Peter Gluckman, the Prime Minister’s Science Adviser, who’s said:

What’s worrying is that there seems to have been a decline in the people represented in the top end of the scale and an increase in the number of people at the bottom end of the scale.

And socioeconomic status may well play a part in this. From the Herald story:

New Zealand was one of just two countries in which socio-economic status had a strong connection to a student’s performance. Some countries’ education systems made up for social disadvantage, but this was not the case in New Zealand.

So any solution addressing the PISA results will of necessity be complex. It’s not going to be sufficient to look only at what’s going on in schools. Yes, support and professional development for STEM teaching across the compulsory sector will be needed. The quality of teaching is definitely important (for a student’s perspective see the Herald article). But without also seriously considering and attempting to deal with the social inequalities in this country, I suspect changes in the educational sector alone will not be enough.

The WordPress Classic Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 164 other followers