Talking Teaching

June 1, 2014

“If you’re going to get lectured at, you might as well be at home in bunny slippers”

This is a post I first wrote for the Bioblog.

There’s an increasing body of literature demonstrating the benefits of active learning for tertiary students taking science subjects. This is a topic I’ve written about before, but I’m always interested in reading more on the subject. And let’s face it, the more evidence the better, when you’re wanting to get lecturers in the sciences engaged in discussion around different ways of teaching. As you’ll have gathered, I find a lot of new science & education material via alerts on Facebook, as well as through the more conventional journal feeds & email alerts, and so it was with this recent paper by Scott Freeman & colleagues, which looks at the effect of active learning on student performance in science, technology, engineering and maths (STEM) classes: I saw it first described in this post1 (whence also comes the quote I’ve used as my title).

The paper by Freeman et al (2014) is a meta-analysis of more than 200 studies of teaching methods used in STEM classes, which included “occasional group problem-solving, worksheets or tutorials completed during class, use of personal response systems with or without peer instruction, and studio or workshop course designs” (ibid.). The impact of the various methods on student learning was measured in two ways: by comparing scores on the same or similar examinations or concept inventories; and by looking at the percentage of students who failed a course.

What did their results show? FIrstly, that students’ mean scores in exams assessing work covered in active learning classes improved by around 6% over more traditional teaching-&-learning formats (& finding that matches those of earlier studies); and secondly, that students in those traditional classes “were 1.5 times more likely to fail”, compared to students given in-class opportunities for active learning (with a ‘raw failure’ rate averaging 33.8% in traditional lecturing classes and 21.8% in more active classes). These results held across all STEM subjects. The researchers also found that active-learning techniques had a stronger effect on concept inventories compared to formal exams (& here I’m wondering if that doesn’t reflect – at least in part – the nature of the exams themselves eg did they give opportunities to demonstrate deep learning?) Interestingly, while the positive impact of active learning was seen across all class sizes, it was more pronounced in classes of less than 50 students.

On the class size thing, I’m wondering if that might be because it’s easier to get everyone actively involved, in a smaller class? For example, I’ve got a colleague at another institution who runs a lot of his classes as ‘flipped’ sessions, and ensures that all students get the opportunity to present to the rest of the group – this is far easier to set up in a class of 50 than in a group with 200+ students in it. (I know! When I run ‘design-a-plant/animal’ sessions, there’s time for only a sub-set of student ‘teams’ to present their creatures to the rest of the class. Plus you really have to work at making sure you get around all teams to talk with them, answer questions, & so on, and so it’s perhaps more likely that someone can remain uninvolved.)

The research team concluded:

Finally, the data suggest that STEM instructors may begin to question the continued use of traditional lecturing in everyday practice, especially in light of recent work indicating that active learning confers disproportionate benefits for STEM students from disadvantaged backgrounds and for female students in male-dominated fields. Although traditional lecturing has dominated undergraduate instruction for most of a millenium and continues to have strong advocates, current evidence suggests that a constructivist “ask, don’t tell” approach may lead to strong increases in student performance, amplifying recent calls from policy-makers and researchers to support faculty who are transforming their STEM courses.

The ‘bunny slippers’ quote from the lead author comes from the post that originally caught my eye. And I suspect there may well be bunny slippers (or the equivalent) in evidence when my own students watch lecture recordings at home :) But this does raise a question around massive open on-line courses (MOOCs), which tend to have a very high ‘fail’ rate – how much of this might be attributed to the difficulty in ensuring opportunities for active learning in these ‘distance’ classes?

And of course, we aren’t really talking a simple dichotomy between ‘traditional’ lecture classes and classes with a very high component of active-learning opportunities – something the research team also note: some of the ‘non-traditional’ methods they surveyed had only a 10-15% ‘active’ component. This is something discussed at more length by Alex Smith in a post entitled “In Defence of the Lecture”. I have to say that his approach sounds very similar to mine, with its mix of socratic questioning, pop quizzes, group discussions, and – yes – sections of ‘lecture’. As Small says:

Not every lecture is a person spending an hour talking nonstop to deliver facts. A good lecture is engaging, it naturally invites discussion and dialogue, it operates at a level much higher than raw information delivery, it is a natural setting for the expert to act as a role model, and it can be integrated with more formal activities (e.g., clicker questions, small-group discussions, etc.).

Lecture should not be the sole means of instruction, and bad lectures are a plague demanding eradication, but we err when we too strenuously inveigh against the lecture.

I couldn’t agree more. And maybe that’s a message that’s being lost in the louder discussion around active learning, and which needs to be heard more widely.

1 The comments thread for this story is also worth reading.

S.Freeman, S.L.Eddy, M.McDonough, M.K.Smith,N.Okorofor, H.Jordt & M.P.Wenderoth  (2014) Active learning increases student performance in science, engineering, and mathematics. http://www.pnas.org/content/early/2014/05/08/1319030111

May 12, 2014

facebook – more than just social networking

Some of my readers over on Sciblogs will probably have realised that I quite like Facebook – not least because it’s a good source of gorgeous images and quirky facts that can start me thinking about a new science blog post. (You don’t see that side of me here on Talking Teaching :D ) Also, it’s fun keeping in contact with friends & participating in various discussion groups.

One of those groups was set up by the biological sciences students at my institution, and it’s used mainly for sharing biology articles and images, the occasional in-joke :) , and alerting other students to upcoming events that their committee has organised. This particular page sees a bit more student activity than some of our paper-specific moodle pages, so for a while now I’ve wondered about the potential of a good Facebook page to be more than ‘just’ a place to hang out and share pictures & stories.

Anyway, recently I had a conversation (on FB, lol) with a couple of fellow Ako Aotearoa Academy members about this potential. It turns out that they both use FB quite extensively in their teaching lives and gave me a lot of helpful hints – along with a very recent paper on this very subject (Dougherty & Andercheck, 2014).

Kevin Dougherty and Brita Andercheck teach a large (around 200 students) introductory sociology class at Baylor University in the US. Like all those with classes of this size (or larger), they recognised that one of the major issues they face is

the tendency for students to feel like anonymous spectators rather than active, collaborative participants

- that is, there’s a real risk that many students will not properly engage with classroom activities, & that their learning will suffer as a result. I’ve written previously about flipped teaching as an example of a technique to increase student engagement (& performance), but with a range of different learning styles among class members, what works for one student won’t necessarily work for another.

So, how do Dougherty & Andercheck use social media to enhance their students’ engagement with the subject, and their achievement (as measured against the learning objectives for the paper)?

The larger a class gets, the harder it can be – even with the best will in the world – get everyone actively involved in discussions, debates and group work during class time. Teachers might try & manage this using a Student Learning Management System (SLMS) like Moodle but again, many students don’t really engage here either. (Certainly that’s been my own experience.)

The authors wondered, what about Facebook? After all,

[s]ocial media, such as Facebook, Twitter, and Instagram, are part of life for the generation of students now filling college classes

and it’s easy to load material and set up discussion threads. (Even a relatively technological illiterate like me can do it!) Why not use it as a more engaging SLMS, one that’s more likely to get buy-in from students because it’s already familiar to them?

I can just hear the cries of horror that might greet such a proposition. Don’t students already spend far too much time on FB and other networking sites? It would just be a distraction. These are valid objections. But with evidence in favour from a developing body of research into such uses of social media, Dougherty & Andercheck set up a study of the impact of a group FB page on students’ engagement & performance in their own class.

For anyone interested in doing likewise, their paper in Teaching Sociology has a very useful description of how the class page is set up & administered. (One of my Academy colleagues has similar pages for MOOCs that he is involved in; due to their size, he has some students help with the admin.) It was run in parallel with their ‘normal’ SLMS, Blackboard, and the latter was where students obtained class handouts & readings. FB was for sharing & discussion; for videos, news stories, & photos; for the ‘Question of the Day’.

For students unable to participate or uncomfortable participating in the classroom discussion, we invited them to add their thoughts and reflections to the conversation on Facebook. We used poll-style questions on the Facebook Group as another means to engage students.

Students readily got involved, ‘liking’ posts, joining discussions, and posting material. Two weeks into the semester, more than half the class had joined the page, and 2/3 were part of it by the end of the paper. To see how all this activity affected learning outcomes, the researchers carried out content analysis of student postings & matched this to performance, and also asked students for feedback via the usual paper appraisals.

The appraisal data showed that half the class visited the FB page on at least a weekly basis, and that the majority were positive about its effect on their experience in the class. While  24% disagreed (ranging from slight to strong disagreement) that it enhanced their experience, Dougherty & Andercheck noted wryly that “it was students who never or rarely used the Facebook Group who disagreed”. Students also felt that the page gave them a stronger sense of belonging in the course, and also that it positively influenced their achievement of the learning objectives.

Of course, the final proof of the pudding is in the eating (sorry, channeling cooking blog here!): was this reflected in actual performance? The researchers found that FB group membership showed a positive correlation to total quiz points and total points. It also had “a marginally significant, positive relationship” with both a student’s total score for the paper and their score in the final exam, and the number of posts someone made was linked to their quiz score.

What’s more, their analysis of the page’s content and their students’ use of the page clearly shows how involved many class members became in discussion. This is a big point for me: I use Moodle in my own class & it’s sometimes a bit sad to see how little real conversation there is about a topic. We might see a question posted, followed by a couple of answers, & then it all dies down again. Would discussions become deeper & more complex in a different, more familiar (&, let’s face it, less clunky) medium? I guess there really is only one way to find out. (And I’ll be making good use of the very helpful hints provided at the end of this thoughtful, and thought-inspiring, paper!)

K.D.Dougherty & B.Andercheck (2014) Using Facebook to Engage Learners in a Large Introductory Course. Teaching Sociology 42(2): 95-104 DOI: 10.1177/0092055X14521022

March 31, 2014

paying it forward

Over the last few weeks I’ve been mentoring a colleague from another institution, helping put together their portfolio for the 2014 Tertiary Teaching Excellence Awards nominations. It’s been a huge amount of work for them, given the need to encapsulate how they meet the award criteria in a total of 8000 words.

At first this looks an unreachable target, but then once you start writing notes and accumulating statements in support, then the problem becomes how to cut the thing down to size. And many people also find it really hard to write about themselves: it sounds like blowing your own trumpet & that can be a difficult thing to do. (Having said that, I know I looked my own finished portfolio & thought, wow! do I really do all that? It was quite affirming, plus the constant reflection was great for my teaching practice.)

So, it was a lot of work for my colleague, who wrote and edited many drafts, solicited supporting comments from students and colleagues, decided on a ‘theme’ to tie it all together, found suitable images – and all the while also carried their usual demanding teaching & admin roles. (I suspect the research may have taken a back seat for a while.) The end result: fascinating reading on a number of levels and a record of excellent teaching in practice (regardless of what happens in the TTEA stakes).

And on the other end of email & phone, I read those drafts, offered other possibilities for investigation/inclusion, proposed many edits (both large & small), found the occasional image, and suggested cuts – you reach a point where you’ve so much personal investment in what you’ve written that you just can’t bear the prospect of removing anything, no matter how the word limit looms over you**.

Yes, that took quite a bit of time at my end too, & I’ve had other colleagues at my institution asking why on earth I would want to take on such a task. But you see, I believe in paying forward: having won one of these awards myself, I feel that I should share what I’ve learned from the process and to help others with tasks like this.

And I’ve made a new friend as well!

 

**(I gather I also provided a calming influence :)  It’s been a great learning experience for me too, as I’ve learned about the cool things someone else is doing to enhance their teaching & their students’ learning experiences.)

March 12, 2014

teaching plant life cycles – trying a different approach

For whatever reason, I find that many students seem to struggle when it comes to learning about plant life cycles. The whole sporophyte/gametophyte, meiosis/mitosis thing really gets them – & that’s even before we start looking at how the life cycle is modified in different groups of plants. Yes, the textbook has lots of diagrams & yes, I’ve always started simple & worked on from there, with opportunity for plenty of questions, but still there are those for whom the topic fails to click. (Not to mention the lecturers in third-year classes, asking whether we really teach this stuff in first-year.) This year the issue’s become even more of a challenge, given that about 2/3 of my large-ish (N>200) didn’t study plants in year 12 at school.

So this year I wondered if it would help if I drew a really basic cycle on the board, as preparation for a more detailed session in the next lecture. I do this in tuts anyway, but not everyone comes to those… And because I use panopto for recording lectures, I needed to think about the best way to do it, because while there are whiteboards in the lecture room they are non-interactive, & the camera doesn’t do a good job of picking up things on a ‘normal’ board. And this is where having a tablet (not an iPad this time; it’s too frustrating when mine won’t communicate properly with the lecture theatre software) comes into it.

This is because, once the tablet’s hooked up to the lecture room system, then anything I might write on its screen (with my spiffy little stylus) is recorded via panopto. And so I left blank slides in my presentation, & drew all over them when we got to that stage, cute little frogs & everything :) (Why frogs? Because we started off with drawing an outline of an animal life cycle, slotting in meiosis & fertilisation, haploid & diploid – with the opportunity to expand on what those terms might mean – before going on to drawing alternation of generations in a very general sense.

Which sounds fine in practice, doesn’t it? Unfortunately, now that I’ve gone & checked the recording, I see that the material on my tablet DIDN’T make it across to panopto, which is downright annoying & obviously I’ve stuffed up somewhere. OK, everyone in the lecture theatre got the benefit of that experience, but those who weren’t, didn’t :( And part of the reason for doing the recordings, is that those who’ve got lecture clashes can catch up later. Mutter mutter mutter.

However, all is not lost. I’m staying later at work for an evening event, so I’ll do a re-record once I can get into a free lecture theatre.

All part of the learning curve – as is the anonymised ‘feedback’ thread I’ve set up on our Moodle page. If the technique helped most students understand the concept of alternation of generations, then I’ll work on doing it better. If it didn’t, well, I guess I need to go back to the drawing board.

February 11, 2014

musings on moocs

I’ve had a few conversations lately around the topic of Massive Open On-line Courses (or MOOCs). These fully on-line courses, which typically have very high enrolments, have become widely available from overseas providers (my own institution recently developed and ran the first such course in New Zealand, which I see is available again this year). If I had time I’d probably do the occasional one for interest (this one on epigenetics caught my eye).

Sometimes the conversations include the question of whether, and how much, MOOCs might contribute to what’s generally known as the ‘universities of the future’. This has always puzzled me a bit, as in their current incarnation most MOOCs don’t carry credit (there are exceptions), so don’t contribute to an actual degree program; they would seem to work better as ‘tasters’ – a means for people to see what a university might have to offer. Depending on their quality, they could also work to encourage young people into becoming more independent learners, regardless of whether they went on to a university – there’s an interesting essay on this issue here. So I thought it would be interesting to look a bit more closely.

Despite the fact that these courses haven’t been around all that long, there’s already quite a bit published about them, including a systematic review of the literature covering the period 2008-2012 by Liyanagunawardena, Adams, & Williams (2013), and a rather entertaining and somewhat sceptical 2013 presentation by Sir John Daniel, (based largely on this 2012 paper).

The term MOOC has only been in use since 2008, when it was first coined for a course offered by the University of Manitoba, Canada (Lianagunawardena et al, 2013). Daniel comments that the philosophy behind early courses like this was one of ‘connectiveness’, such that resources were freely available to anyone, with learning shared by all those in the course. This was underpinned by the use of RSS feeds, Moodle discussions, blogs, Second Life, & on-line meetings. He characterises ‘modern’ MOOCs as bearing little relation, in their educational philosophy, to these early programs, viewing programs offered by major US universities as

basically learning resources with some computerised feedback. In terms of pedagogy their quality varies widely, from very poor to OK.

Part of the problem here lies with the extremely large enrolments in today’s MOOCs, whereas those early courses were small enough that some semi-individualised interactions between students and educators were possible. Unfortunately the combination of variable pedagogy plus little in the way of real interpersonal interactions in these huge classes also sees them with very high drop-out rates: Liyanawardena and her colleagues note that the average completion rate is less than 10% of those beginning a course, with the highest being 19.2% for a Coursera offering.

Daniel offers some good advice to those considering setting up MOOCs of their own, given that currently – in his estimation – there are as yet no good business models available for these courses. Firstly: don’t rush into it just because others are. Secondly,

have a university-wide discussion on why you might offer a MOOC or MOOCs and use it to develop a MOOC strategy. The discussion should involve all staff members who might be involved in or affected by the offering of a MOOC.

His third point: ensure that any MOOC initiatives are fully integrated into your University’s strategy for online learning (my emphasis). To me this is an absolute imperative – sort the on-line learning strategy first, & then consider how MOOCs might contribute to this. (Having said that, I notice that the 2014 NMC Horizon report on higher education, by Johnson et al.,  sees these massive open on-line courses as in competition with the universities, rather than complementary to their on-campus and on-line for-credit offerings. And many thanks to Michael Edmonds for the heads-up on this paper.)

This is in fact true for anything to do with moving into the ‘universities of the future space (with or without MOOCs). Any strategy for online learning must surely consider resourcing: provision not only of the hardware, software, and facilities needed to properly deliver a ‘blended’ curriculum that may combine both face-to-face and on-line delivery, but also of the professional development needed to ensure that educators have the pedagogical knowledge and skills to deliver excellent learning experiences and outcomes in what for most of us is a novel environment. For there’s far more to offering a good on-line program than simply putting the usual materials up on a web page. A good blended learning (hybrid) system must be flexible, for example; it must suit

the interests and desires of students, who are able to choose how they attend lecture – from the comfort of their home, or face-to-face with their teachers. Additionally, … students [feel] the instructional technology [makes] the subject more interesting, and increase[s] their understanding, as well as encourag[ing] their participation… (Johnson et al., 2014).

This is something that is more likely to encourage the sort of critical thinking and deep learning approaches that we would all like to see in our students.

Furthermore, as part of that hybridisation, social media are increasingly likely to be used in learning experiences as well as for the more established patterns of social communication and entertainment (eg Twitter as a micro-blogging tool: Liyanagunawardena et al., 2013). In fact, ‘external’ communications (ie outside of learning management systems such as Moodle) are likely to become more significant as a means of supporting learner groups in this new environment – this is something I’m already seeing with the use of Facebook for class discussions and sharing of ideas and resources. Of course, this also places demands on educators:

Understanding how social media can be leveraged for social learning is a key skill for teachers, and teacher training programs are increasingly being expected to include this skill. Understanding how social media can be leveraged for social learning is a key skill for teachers, and teacher training programs are increasingly being expected to include this skill. (Johnson et al., 2014).

There is also a need, in any blended learning system, to ensure skilled moderation of forums and other forms of on-line engagement, along with policies to ensure privacy is maintained and bullying and other forms of unacceptable behaviour are avoided or nipped in the bud (Liyanawardena et al. 2013; Johnson et al., 2014). And of course there’s the issue of flipped classrooms, something that the use of these technologies really encourages but which very few teaching staff have any experience of.

Another issue examined by Liyanagunawardena and her colleagues, in their review of the MOOC literature, is that of digital ‘natives': are our students really able to use new learning technologies in the ways that we fondly imagine they can? This is a question that applies just as well to the hybrid learning model of ‘universities of the future’. In one recent study cited by the team, researchers found that of all the active participants in a particular MOOC, only one had never been involved in other such courses. This begs the question of “whether a learner has to learn how to learn” in the digital, on-line environment. (Certainly, I’ve found I need to show students how to download podcasts of lectures, something I’d naively believed that they would know how to do better than I!) In other words, any planning for blended delivery must allow for helping learners, as well as teachers, to become fluent in the new technologies on offer.

We live in interesting times.

And I would love to hear from any readers who have experience in this sort of learning environment.

T.R.Liyanagunawardena, A.A.Adams & S.A.Williams (2013) MOOCs: a systematic study of the published literature 2008-2012. The International Review of Research in Open and Distance Learning 14(3): 202-227

L.Johnson, S.Adams Becker, V.Estrada, & A.Freeman (2014) NMC HOrizon Report: 2014 Higher Education Edition. Austin, Texas. The New Media Consortium. ISBN 978-0-9897335-5-7

January 21, 2014

teaching laboratories – the shape of things to come?

A quick post from notes I took during another talk at the Ako Aotearoa Symposium last December: this was an exciting presention on the changing form of teaching laboratories, by Ken Collins and Joanne Kelly from Labworks Architecture (another colleague also mentioned this session, in her own post on the day’s proceedings).  Ken & Joanne began by noting that lab spaces are used for students to gain and enhance a range of skills: critical thinking, developing solutions to problems, working collaboratively, practising practical skills. ‘Traditional’ lab spaces don’t really accomodate all this, they said, & went on to explain why & to share with us some of the solutions they’ve developed for various clients.

Their focus was on the links between space, technology, and pedagogy (something that’s been missing in most of the labs I’ve taught in, where the technology’s been retrofitted as need and funding dictate). Having more flexible spaces encourages pedagogy, which in turn is enabled by space. Pedagogy is enhanced by technology, which will also place demands on space – after all, if you’re using computer screens to show things, you want to be in a room where all students have a clear line of sight to the sceens. In other words, a modern teaching space embeds technology, which of course extends how we use the space. (I see this a lot in the way our wonderful first-year tutor delivers our lab classes, retrofitted technology & all.)

More & more, this is equally true for how we use lecture/tutorial spaces.

‘Old-style’ learning spaces have always tended to focus on the perceived needs of the teacher, & to support highly structured, teacher-led, ‘instructional’ (didactic) learning experiences. Joanne & Ken believe – & I think most of those who attended their presentation would agree – that these days, in a modern classroom, about 15% of lab-room learning would be teacher-led. Of the remainder around would see students collaborating on various investigations 75% – ie there’s much more collaborative problem-solving, which realistically is how many workplaces operate anyway – and the remaining time is given over to small- & large-group discussion & feedback. It’s arguable whether that’s best done in a lab, & so the presenters showed classrooms they’ve designed where glass doors separate formal lab space from breakout spaces. I immediately added that to my mental ‘I’d really like this for our students’ list :)

They concluded by asking us to think about classroom space in general. We’re already seeing a move from libraries as study environment to ‘hubs’, with individual work spaces alongside commons, cafes, and alcoves where people can chill out or just sit for a quiet discussion. What will the future be like, as we continue down this road? (More virtual reality, perhaps? At a previous symposium we heard about the use of ‘virtual labs’, for example, via Second Life, allowing students to practice lab skills & protocols before actually coming into the real-world lab.) Certainly any changes should allow & support innovative practice in teaching & learning; for example, new lecture theatres could be low-pitched rather than steep, with room to move between rows, & thoroughly technology-enabled.

I’ll have to make sure these options are on the table, when the time for lab refurbishment rolls round.

December 13, 2013

can we help students too much?

One of the (many) good things about writing exams at a national level was that it really taught me how to write a good question: one that lacks ambiguity, clearly identifies what’s being asked of the students, and gives them opportunities to demonstrate their knowledge and understanding. So I was a bit saddened when marking exams recently to find that one question, in particular, wasn’t eliciting what I’d hoped for.

You see, I’d written a question in two parts. Students were first asked to summarise their knowledge the process of gene expression, and then – using the phrase “in your answer you should also” – asked to explain how various events might introduce diversity into the genome. And for some reason many students went straight past the first part and answered the second (and generally did this very well, I might add).

It’s the first time I’ve seen this to any marked degree and it really bothered me, & I’ve talked about it at some length with a colleague. We’re beginning to wonder if maybe we give students too much information around the exams. Now that may sound wrong – you’d think that demystifying exams, in the sense of giving access to previous papers so students get an idea of what to expect, would be helpful. It also highlights key themes in the subject that we return to again & again. We don’t just leave it at this, mind you: we spend time in tuts discussing things like how to ‘unpackage’ questions and how to plan the answer before actually writing it. But we’re wondering if making previous papers readily available in the study guide is leading to students ‘picking questions’ – deciding what’s likely to be asked, on the basis of what’s been asked in the past, and then not just studying around that but perhaps writing and learning ‘model’ answers off by heart. And regurgitating that answer regardless of what the question actually requires of them.

And that sort of learning is not the sort of learning I want to encourage.

So we decided to do things a little differently with next year’s study guide & include just a single year’s exam paper in it. Students will still be able to access many more through the library’s database; they’ll just have to do a bit more work themselves. And thinking more on this as I’m writing, perhaps we should also devote a bit of tut time to explicitly recognising those themes & working with the students to build up a ‘big picture’ view of the main ideas associated with them, by way of encouraging that breadth of understanding – and emphasising that this approach should allow them to approach with confidence any question we might ask.

Your thoughts?

December 12, 2013

Evaluating teaching the hard-nosed numbers way

[This is a copy of a post on my blog PhysicsStop, sci.waikato.ac.nz/physicsstop, 10 December 2013]

Recently there’s been a bit of discussion in our Faculty on how to get a reliable evaluation of people’s teaching. The traditional approach is with the appraisal. At the end of each paper the students get to answer various questions on the teacher’s performance on a five-point Likert Scale (i.e. ‘Always’, ‘Usually’, ‘Sometimes’, ‘Seldom’, ‘Never’.)  For example: “The teacher made it clear what they expected of me.” The response ‘Always’ is given a score of 1, ‘Usually’ is given 2, down to ‘Never’ which is given a score of 5. An averaged response of the questions across students gives some measure of teaching success – ranging in theory from 1.0 (perfect) through to 5.0 (which we really, really don’t want to see happening).

We’ve also got a general question – “Overall, this teacher was effective”. This is also given a score on the same scale.

A question that’s been raised is: Does the “Overall, this teacher was effective” score correlate well with the average of the others?

I’ve been teaching for several years now, and have a whole heap of data to draw from. So, I’ve been analyzing it (for 2008 onwards), and, in the interests of transparency, I’m happy for people to see it.  For myself, the question of “does a single ‘overall’ question get a similar mark to the averaged response of the other questions?” is a clear yes. The graph below shows the two scores plotted against each other, for different papers that I have taught. For some papers I’ve had a perfect score – 1.0 by every student for every question. For a couple scores have been dismall (above 2 on average):

Capture1.JPG

What does this mean? That’s a good question. Maybe it’s simply that a single question is as good as a multitude of questions if all we are going to do is to take the average of something. More interesting is to look at each question in turn. The questions start with “the teacher…” and then carry on as in the chart below, which shows the responses I’ve had averaged over papers and years.
Capture2.JPG
Remember, low scores are good. And what does this tell me? Probably not much that I don’t already know. For example, anecdotally at any rate, the question “The teacher gave me helpful feedback” is a question for which many lecturers get their poorest scores (highest numbers). This may well be because students don’t realize they are getting feedback. I have colleagues who, when they give oral feedback, will prefix what they say with “I am now giving you feedback on how you have done” so that it’s recognized for what it is.
So, another question. How much have I improved in recent years? Surely I am a better teacher now than what I was in 2008. I really believe that I am. So my scores should be heading towards 1.  Well, um, maybe not. Here they are. There are two lines – the blue line is the response to the question ‘Overall, this teacher was effective’, averaged over all the papers I took in a given year; the red line is the average of the other questions, averaged over all the papers. The red line closely tracks the blue – this shows the same effect as seen on the first graph. The two correlate well.
Capture3.JPG
So what’s happening. I did something well around 2010 but since then it’s gone backwards (with a bit of a gain this year – though not all of this year’s data has been returned to me yet). There are a couple of comments to make. In 2010 I started on a Post Graduate Certificate of Tertiary Teaching. I put a lot of effort into this. There were a couple of major tasks that I did that were targeted at implementing and assessing a teaching intervention to improve student performance. I finished the PGCert in 2011. That seems to have helped with my scores, in 2010 at least. A quick peruse of my CV, however, will tell you that this came at the expense of research outputs. Not a lot of research was going on in my office or lab during that time.  And what happened in 2012? I had a period of study leave (hooray for research outputs!) followed immediately by a period of parental leave. Unfortunately, I had the same amount of teaching to do and that got squashed into the rest of the year. Same amount of material, less time to do it, poorer student opinions. It seems a logical explanation anyway.
Does all this say anything about whether I am an effective teacher? Can one use a single number to describe it? These are questions that are being considered. Does my data help anyone to answer these questions? You decide.

December 9, 2013

shaking up the academy? or how the academy could shake up teaching

Last week I spent a couple of days down in Wellington, attending the annual symposium for the Ako Aotearoa Academy. The Academy’s made up of the winners of the national Tertiary Teaching Excellence Awards, so there are around 150 or so of us now. While only 35 members were able to make it to this year’s event (& the executive committee will survey everyone to see if there’s a better time – having said that, everyone seems so busy that there’s probably no date that would suit everyone!), we had a great line-up of speakers & everyone left feeling inspired & energised. I’ll blog about several of those presentations, but thought I would start with one by Peter Coolbear, who’s the director of our parent body, Ako Aotearoa.

Peter began by pointing out that the Academy is potentially very influential – after all, it’s made up of tertiary teachers recognised at the national level for the quality of their teaching, & who foster excellence in learning & teaching at their own institutions.  But he argued – & I agree with him – that there is room for us to become involved in the wider scene. Peter had a number of suggestions for us to consider.

First up, there’s a lot going on in the area of policy – are there areas where the Academy might be expected to have & express an opinion? For example

  • There’s the latest draft of the Tertiary Education Strategy (TES), which “sets out the Government’s long-term strategic direction for tertiary education; and its current and medium-term priorities for tertiary education.” There’s a link to the Minister’s speech announcing the launch of the draft strategy here.
  • In addition, the State Services Commission’s document Better Public Services: results for New Zealanders sets out 10 targets across 5 areas. Targets 5 & 6 are relevant here as they are a reference point for government officials looking at evidence for success in the education sector. (Such scrutiny is likely to become more intense in light of the 2012 PISA results, which have just been made public.) Target 5 expects that we’ll “[increase] the proportion of 18-year-olds with NCEA level 2 or equivalent qualification”; #6 is looking for an increase in ” the proportion of 25 to 34-year-olds with advanced trade qualifications, diplomas and degrees (at level 4 or above)”. This will increase the pressure on institutions to increase retention & completion rates – might this have an effect on standards?
  • There’s also the requirement to achieve parity of success for ‘priority’ learners, especially Maori & Pasifika – this is priority #3 in the TES. (Kelly Pender, from Bay of Plenty Polytech, gave an inspirational presentation on how he weaves kaupapa Maori into pretty much everything he does in his classroom, in an earlier session.) And it’s an important one for us to consider. Peter cited data from the Ministry of Education’s website, ‘Education Counts’, which showed significantly lower completion rates for Maori & Pasifika students in their first degrees compared to European students, and commented that this will likely become a major issue for the universities in the near future.
  • If we’re to meet those achievement requirements, then how institutions scaffold learners into higher-level study, through foundation & transition programs, will become increasingly important. What are the best ways to achieve this?
  • Peter predicted increased accountability for the university sector (including governance reform). Cycle 5 of NZ’s Academic Audits has begun, and “is to be framed around academic activities related to teaching and learning and student support.” This is definitely one I’d expect Academy members to have an opinion on!
  • He also expects strengthened quality assurance processes throughout the education sector: this suggests a stronger (& more consistent) role for the  NZ Qualifications Authority, with the development of partnership dialogues across the sector (ie including universities).

Then, at the level of the providers (ie the educational institutions themselves – & that’s not just the polytechs & universities), we have:

  • a targeted review of qualifications offered at pre-degree level – there’s background information here;
  • a government-led drive to get more learners into the ‘STEM’ subjects (science, technology, engineering, & maths) – this poses some interesting challenges as, at university level, we’re seeing quite a few students who’ve not taken the right mix of subjects, at the right NCEA level, to go directly into some of the STEM papers they need for, say, an engineering degree;
  • the rise in Massive Open On-line Courses, or MOOCs. (I find these quite strange creatures as they are free to the student and typically attract very large enrolments, but also apparently have very low completion rates. What’s in them for the institution? A good way of offering ‘taster’ courses that hook students in?)
  • the likelihood that we will see the development of a system for professional accreditation of tertiary teachers (I’ve written about this previously and will write another post fairly soon, as accreditation was the subject of a thought-provoking session at the symposium);
  • how we achieve protection of academic standards – it’s possible that government policies (eg those linking funding to completion & retention rates) may result in a tendency to exclude of underprepared kids &/or lowering standards – neither is desirable but both are possible results of those policies.

That’s a big list and the Academy can’t do everything! So, what should it focus on? (This is not a rhetorical question – it would be great to get some discussion going.) The Academy, in the person of its members, is effectively a resource; a body of expertise – can it become a ‘go-to’ body for advice? Speaking personally I think we need to make that shift; otherwise we remain invisible outside our individual institutions & the teaching-focused activities we’re involved in, & in a politicised world that’s not a comfortable thing to be. Can we, for example, better promote the significance of teaching excellence outside the education sector? Become involved in the discussions around & development of any accreditation scheme? Develop position papers around maintaining teaching excellence in the context of the new TES?

What do you think? And what shall we, collectively, do about it?

November 21, 2013

learning leadership

and yes, that’s an intentionally ambiguous title :) (The full version was Learning leadership: the interplay between our own professional development and our classroom practice.)

I recently gave a pecha kucha** presentation on this subject, at an Ako Aotearoa mini-symposium up in Auckland. The idea for the subject of my presentation leapt to the front of my mind while I was at a Teaching Network*** meeting looking at how to raise the profile of teaching in tertiary institutions (specifically, universities). One of my colleagues kicked off part of the discussion with a brief talk on developing leadership in teaching, & I thought, all this applies to leading/guiding our students to become better learners, as well. Which is pretty much the thrust of my presentation. I pretty much use slides as talking points:

Slide2

 

Slide3

Slide4

Slide5

Slide6

Slide7

Slide8

Slide9

Slide10

And we finished up with some ideas on what future-focused leadership in teaching and learning could look like.

It would be really good to hear your thoughts on this :)

 

 

** a maximum of 20 slides, with a maximum of 20 seconds per slide. Certainly forces you to focus your ideas. Mine wasn’t that long, because I wanted to use it to spur discussion & so we need time for that in my short presentation slot.

*** an in-house group for staff from across the institution with an interest in all things to do with teaching.

Older Posts »

The WordPress Classic Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 164 other followers