Talking Teaching

June 1, 2014

“If you’re going to get lectured at, you might as well be at home in bunny slippers”

This is a post I first wrote for the Bioblog.

There’s an increasing body of literature demonstrating the benefits of active learning for tertiary students taking science subjects. This is a topic I’ve written about before, but I’m always interested in reading more on the subject. And let’s face it, the more evidence the better, when you’re wanting to get lecturers in the sciences engaged in discussion around different ways of teaching. As you’ll have gathered, I find a lot of new science & education material via alerts on Facebook, as well as through the more conventional journal feeds & email alerts, and so it was with this recent paper by Scott Freeman & colleagues, which looks at the effect of active learning on student performance in science, technology, engineering and maths (STEM) classes: I saw it first described in this post1 (whence also comes the quote I’ve used as my title).

The paper by Freeman et al (2014) is a meta-analysis of more than 200 studies of teaching methods used in STEM classes, which included “occasional group problem-solving, worksheets or tutorials completed during class, use of personal response systems with or without peer instruction, and studio or workshop course designs” (ibid.). The impact of the various methods on student learning was measured in two ways: by comparing scores on the same or similar examinations or concept inventories; and by looking at the percentage of students who failed a course.

What did their results show? FIrstly, that students’ mean scores in exams assessing work covered in active learning classes improved by around 6% over more traditional teaching-&-learning formats (& finding that matches those of earlier studies); and secondly, that students in those traditional classes “were 1.5 times more likely to fail”, compared to students given in-class opportunities for active learning (with a ‘raw failure’ rate averaging 33.8% in traditional lecturing classes and 21.8% in more active classes). These results held across all STEM subjects. The researchers also found that active-learning techniques had a stronger effect on concept inventories compared to formal exams (& here I’m wondering if that doesn’t reflect – at least in part – the nature of the exams themselves eg did they give opportunities to demonstrate deep learning?) Interestingly, while the positive impact of active learning was seen across all class sizes, it was more pronounced in classes of less than 50 students.

On the class size thing, I’m wondering if that might be because it’s easier to get everyone actively involved, in a smaller class? For example, I’ve got a colleague at another institution who runs a lot of his classes as ‘flipped’ sessions, and ensures that all students get the opportunity to present to the rest of the group – this is far easier to set up in a class of 50 than in a group with 200+ students in it. (I know! When I run ‘design-a-plant/animal’ sessions, there’s time for only a sub-set of student ‘teams’ to present their creatures to the rest of the class. Plus you really have to work at making sure you get around all teams to talk with them, answer questions, & so on, and so it’s perhaps more likely that someone can remain uninvolved.)

The research team concluded:

Finally, the data suggest that STEM instructors may begin to question the continued use of traditional lecturing in everyday practice, especially in light of recent work indicating that active learning confers disproportionate benefits for STEM students from disadvantaged backgrounds and for female students in male-dominated fields. Although traditional lecturing has dominated undergraduate instruction for most of a millenium and continues to have strong advocates, current evidence suggests that a constructivist “ask, don’t tell” approach may lead to strong increases in student performance, amplifying recent calls from policy-makers and researchers to support faculty who are transforming their STEM courses.

The ‘bunny slippers’ quote from the lead author comes from the post that originally caught my eye. And I suspect there may well be bunny slippers (or the equivalent) in evidence when my own students watch lecture recordings at home :) But this does raise a question around massive open on-line courses (MOOCs), which tend to have a very high ‘fail’ rate – how much of this might be attributed to the difficulty in ensuring opportunities for active learning in these ‘distance’ classes?

And of course, we aren’t really talking a simple dichotomy between ‘traditional’ lecture classes and classes with a very high component of active-learning opportunities – something the research team also note: some of the ‘non-traditional’ methods they surveyed had only a 10-15% ‘active’ component. This is something discussed at more length by Alex Smith in a post entitled “In Defence of the Lecture”. I have to say that his approach sounds very similar to mine, with its mix of socratic questioning, pop quizzes, group discussions, and – yes – sections of ‘lecture’. As Small says:

Not every lecture is a person spending an hour talking nonstop to deliver facts. A good lecture is engaging, it naturally invites discussion and dialogue, it operates at a level much higher than raw information delivery, it is a natural setting for the expert to act as a role model, and it can be integrated with more formal activities (e.g., clicker questions, small-group discussions, etc.).

Lecture should not be the sole means of instruction, and bad lectures are a plague demanding eradication, but we err when we too strenuously inveigh against the lecture.

I couldn’t agree more. And maybe that’s a message that’s being lost in the louder discussion around active learning, and which needs to be heard more widely.

1 The comments thread for this story is also worth reading.

S.Freeman, S.L.Eddy, M.McDonough, M.K.Smith,N.Okorofor, H.Jordt & M.P.Wenderoth  (2014) Active learning increases student performance in science, engineering, and mathematics. http://www.pnas.org/content/early/2014/05/08/1319030111

May 12, 2014

facebook – more than just social networking

Some of my readers over on Sciblogs will probably have realised that I quite like Facebook – not least because it’s a good source of gorgeous images and quirky facts that can start me thinking about a new science blog post. (You don’t see that side of me here on Talking Teaching :D ) Also, it’s fun keeping in contact with friends & participating in various discussion groups.

One of those groups was set up by the biological sciences students at my institution, and it’s used mainly for sharing biology articles and images, the occasional in-joke :) , and alerting other students to upcoming events that their committee has organised. This particular page sees a bit more student activity than some of our paper-specific moodle pages, so for a while now I’ve wondered about the potential of a good Facebook page to be more than ‘just’ a place to hang out and share pictures & stories.

Anyway, recently I had a conversation (on FB, lol) with a couple of fellow Ako Aotearoa Academy members about this potential. It turns out that they both use FB quite extensively in their teaching lives and gave me a lot of helpful hints – along with a very recent paper on this very subject (Dougherty & Andercheck, 2014).

Kevin Dougherty and Brita Andercheck teach a large (around 200 students) introductory sociology class at Baylor University in the US. Like all those with classes of this size (or larger), they recognised that one of the major issues they face is

the tendency for students to feel like anonymous spectators rather than active, collaborative participants

- that is, there’s a real risk that many students will not properly engage with classroom activities, & that their learning will suffer as a result. I’ve written previously about flipped teaching as an example of a technique to increase student engagement (& performance), but with a range of different learning styles among class members, what works for one student won’t necessarily work for another.

So, how do Dougherty & Andercheck use social media to enhance their students’ engagement with the subject, and their achievement (as measured against the learning objectives for the paper)?

The larger a class gets, the harder it can be – even with the best will in the world – get everyone actively involved in discussions, debates and group work during class time. Teachers might try & manage this using a Student Learning Management System (SLMS) like Moodle but again, many students don’t really engage here either. (Certainly that’s been my own experience.)

The authors wondered, what about Facebook? After all,

[s]ocial media, such as Facebook, Twitter, and Instagram, are part of life for the generation of students now filling college classes

and it’s easy to load material and set up discussion threads. (Even a relatively technological illiterate like me can do it!) Why not use it as a more engaging SLMS, one that’s more likely to get buy-in from students because it’s already familiar to them?

I can just hear the cries of horror that might greet such a proposition. Don’t students already spend far too much time on FB and other networking sites? It would just be a distraction. These are valid objections. But with evidence in favour from a developing body of research into such uses of social media, Dougherty & Andercheck set up a study of the impact of a group FB page on students’ engagement & performance in their own class.

For anyone interested in doing likewise, their paper in Teaching Sociology has a very useful description of how the class page is set up & administered. (One of my Academy colleagues has similar pages for MOOCs that he is involved in; due to their size, he has some students help with the admin.) It was run in parallel with their ‘normal’ SLMS, Blackboard, and the latter was where students obtained class handouts & readings. FB was for sharing & discussion; for videos, news stories, & photos; for the ‘Question of the Day’.

For students unable to participate or uncomfortable participating in the classroom discussion, we invited them to add their thoughts and reflections to the conversation on Facebook. We used poll-style questions on the Facebook Group as another means to engage students.

Students readily got involved, ‘liking’ posts, joining discussions, and posting material. Two weeks into the semester, more than half the class had joined the page, and 2/3 were part of it by the end of the paper. To see how all this activity affected learning outcomes, the researchers carried out content analysis of student postings & matched this to performance, and also asked students for feedback via the usual paper appraisals.

The appraisal data showed that half the class visited the FB page on at least a weekly basis, and that the majority were positive about its effect on their experience in the class. While  24% disagreed (ranging from slight to strong disagreement) that it enhanced their experience, Dougherty & Andercheck noted wryly that “it was students who never or rarely used the Facebook Group who disagreed”. Students also felt that the page gave them a stronger sense of belonging in the course, and also that it positively influenced their achievement of the learning objectives.

Of course, the final proof of the pudding is in the eating (sorry, channeling cooking blog here!): was this reflected in actual performance? The researchers found that FB group membership showed a positive correlation to total quiz points and total points. It also had “a marginally significant, positive relationship” with both a student’s total score for the paper and their score in the final exam, and the number of posts someone made was linked to their quiz score.

What’s more, their analysis of the page’s content and their students’ use of the page clearly shows how involved many class members became in discussion. This is a big point for me: I use Moodle in my own class & it’s sometimes a bit sad to see how little real conversation there is about a topic. We might see a question posted, followed by a couple of answers, & then it all dies down again. Would discussions become deeper & more complex in a different, more familiar (&, let’s face it, less clunky) medium? I guess there really is only one way to find out. (And I’ll be making good use of the very helpful hints provided at the end of this thoughtful, and thought-inspiring, paper!)

K.D.Dougherty & B.Andercheck (2014) Using Facebook to Engage Learners in a Large Introductory Course. Teaching Sociology 42(2): 95-104 DOI: 10.1177/0092055X14521022

April 25, 2014

plagiarism & managing it

Filed under: education, university — Tags: , , , , , — alison @ 10:14 pm

I’m marking first-year essays at the moment. Because these students have had little or no practice at writing scientific essays before they arrive in my class, we give them a lot of learning support. There’s a marking rubric, which students get along with the questions at the very beginning of the semester. (Alas! This doesn’t seem to stop the last-minute rush-combined-with-sheer-panic!) We spend time on tuts discussing how to structure an essay, how to cite and to reference & to paraphrase, and so on. Both the senior tutor & I are more than happy to comment on drafts – some of my colleagues think we’re nuts, but giving formative feedback early in the piece significantly improves the final essay & means less time is spent at that end. And this year I followed the example of my friend Margaret Henley and ran a drop-in session in the student centre: I was there, along with the Science librarian and staff from Student Learning, and the 50 or so students who attended moved around between us depending on what they needed. (Far more time-efficient then having the same number of students turn up to see us in drips and drabs.

And of course we also discuss at some length the issues and concepts relating to plagiarism, and students’ essays are put through the Turnitin system on submission. (This year I set it up so that they could see their score after submission, which they seem to quite like.) So I was interested to see this story on plagiarism and cheating in NZ universities,  in the NZ Herald a few days ago. It was notable that there was a bit of variation between institutions in the number of instances of cheating that were detected, which I suspect has more to do with processes than with actual differences in (dis)honesty in the student bodies. We all seem to handle it differently, too; my own institution has a student discipline committee, to which all instances of suspected plagiarism are supposed to be referred. I like this system – it is a lot more transparent in that the paper convenor doesn’t end up being the judge, jury, and executioner (with all the potential conflicts that this entails), and more consistent because the same set of standards, and outomes, is applied across the board. Which is probably why I felt more than a little uncomfortable to see that, in one instance reported in the Herald story, an individual lecturer seemed to be making the judgement call. Maybe that was just the way the story came across in the paper. I hope so.

There’s an interesting discussion here on why students plagiarise, which suggests that maybe we, the teachers, have something to do with it in that we maybe don’t do enough to help our students develop their own ‘voice’ and the confidence to use it:

Students… often stumble into plagiarism (or rush head-long into it) because they either cannot find or do not trust the authority of their own voice.

The author, Nick Carbone, concludes that

[h]elping students find their own voice, their own words, so that they can distinguish better their voices and words from the voices and words of the sources they research, hear, read, and that really, when you think about it, always already surround them, seems to me more and more, the best way to help students understand, really, what plagiarism is all about.

I’m not sure how feasible it actually is, in a paper that’s not first & foremost a writing paper, to help all students find their ‘voice’. (Nor am I sure that all academics would view it as part of their role to do so.) And I definitely agree with Jonathan Bailey that the ultimate responsibility for plagiarism does rest with the student. But – as he says – teachers can do a lot both to educate students about academic integrity and to minimise the temptation and the pressure to plagiarise. For example, the pressures involved around large high-stakes assignments may make a spot of cheating look more attractive. Bailey lists the following steps to reduce plagiarism’s allure (but also reminds us that the problem’s never going to go away completely):

  1. Educate on Plagiarism: Teach students clearly what it is and how to avoid it. Discuss plagiarism openly and without scare tactics.

  2. Craft Plagiarism-Resistant Assignments: Use prompts that can’t be Googled, require multiple drafts and include in-class portions when possible.

  3. Connect With Students: Offer to help and give students the support they need so they are confident they can complete the assignment.

  4. Forgive Mistakes: Understand that mistakes happen and treat them as chances to teach, not discipline.

  5. Discipline Fairly: Those who clearly are trying to cheat should be disciplined fairly and strongly as appropriate.

Which makes me feel that we’re doing something right, in my first-year papers. (It also reminds me how frustrated I get to see the same questions pop up in tests and exams, year after year. What do people expect?!)

March 31, 2014

paying it forward

Over the last few weeks I’ve been mentoring a colleague from another institution, helping put together their portfolio for the 2014 Tertiary Teaching Excellence Awards nominations. It’s been a huge amount of work for them, given the need to encapsulate how they meet the award criteria in a total of 8000 words.

At first this looks an unreachable target, but then once you start writing notes and accumulating statements in support, then the problem becomes how to cut the thing down to size. And many people also find it really hard to write about themselves: it sounds like blowing your own trumpet & that can be a difficult thing to do. (Having said that, I know I looked my own finished portfolio & thought, wow! do I really do all that? It was quite affirming, plus the constant reflection was great for my teaching practice.)

So, it was a lot of work for my colleague, who wrote and edited many drafts, solicited supporting comments from students and colleagues, decided on a ‘theme’ to tie it all together, found suitable images – and all the while also carried their usual demanding teaching & admin roles. (I suspect the research may have taken a back seat for a while.) The end result: fascinating reading on a number of levels and a record of excellent teaching in practice (regardless of what happens in the TTEA stakes).

And on the other end of email & phone, I read those drafts, offered other possibilities for investigation/inclusion, proposed many edits (both large & small), found the occasional image, and suggested cuts – you reach a point where you’ve so much personal investment in what you’ve written that you just can’t bear the prospect of removing anything, no matter how the word limit looms over you**.

Yes, that took quite a bit of time at my end too, & I’ve had other colleagues at my institution asking why on earth I would want to take on such a task. But you see, I believe in paying forward: having won one of these awards myself, I feel that I should share what I’ve learned from the process and to help others with tasks like this.

And I’ve made a new friend as well!

 

**(I gather I also provided a calming influence :)  It’s been a great learning experience for me too, as I’ve learned about the cool things someone else is doing to enhance their teaching & their students’ learning experiences.)

February 11, 2014

musings on moocs

I’ve had a few conversations lately around the topic of Massive Open On-line Courses (or MOOCs). These fully on-line courses, which typically have very high enrolments, have become widely available from overseas providers (my own institution recently developed and ran the first such course in New Zealand, which I see is available again this year). If I had time I’d probably do the occasional one for interest (this one on epigenetics caught my eye).

Sometimes the conversations include the question of whether, and how much, MOOCs might contribute to what’s generally known as the ‘universities of the future’. This has always puzzled me a bit, as in their current incarnation most MOOCs don’t carry credit (there are exceptions), so don’t contribute to an actual degree program; they would seem to work better as ‘tasters’ – a means for people to see what a university might have to offer. Depending on their quality, they could also work to encourage young people into becoming more independent learners, regardless of whether they went on to a university – there’s an interesting essay on this issue here. So I thought it would be interesting to look a bit more closely.

Despite the fact that these courses haven’t been around all that long, there’s already quite a bit published about them, including a systematic review of the literature covering the period 2008-2012 by Liyanagunawardena, Adams, & Williams (2013), and a rather entertaining and somewhat sceptical 2013 presentation by Sir John Daniel, (based largely on this 2012 paper).

The term MOOC has only been in use since 2008, when it was first coined for a course offered by the University of Manitoba, Canada (Lianagunawardena et al, 2013). Daniel comments that the philosophy behind early courses like this was one of ‘connectiveness’, such that resources were freely available to anyone, with learning shared by all those in the course. This was underpinned by the use of RSS feeds, Moodle discussions, blogs, Second Life, & on-line meetings. He characterises ‘modern’ MOOCs as bearing little relation, in their educational philosophy, to these early programs, viewing programs offered by major US universities as

basically learning resources with some computerised feedback. In terms of pedagogy their quality varies widely, from very poor to OK.

Part of the problem here lies with the extremely large enrolments in today’s MOOCs, whereas those early courses were small enough that some semi-individualised interactions between students and educators were possible. Unfortunately the combination of variable pedagogy plus little in the way of real interpersonal interactions in these huge classes also sees them with very high drop-out rates: Liyanawardena and her colleagues note that the average completion rate is less than 10% of those beginning a course, with the highest being 19.2% for a Coursera offering.

Daniel offers some good advice to those considering setting up MOOCs of their own, given that currently – in his estimation – there are as yet no good business models available for these courses. Firstly: don’t rush into it just because others are. Secondly,

have a university-wide discussion on why you might offer a MOOC or MOOCs and use it to develop a MOOC strategy. The discussion should involve all staff members who might be involved in or affected by the offering of a MOOC.

His third point: ensure that any MOOC initiatives are fully integrated into your University’s strategy for online learning (my emphasis). To me this is an absolute imperative – sort the on-line learning strategy first, & then consider how MOOCs might contribute to this. (Having said that, I notice that the 2014 NMC Horizon report on higher education, by Johnson et al.,  sees these massive open on-line courses as in competition with the universities, rather than complementary to their on-campus and on-line for-credit offerings. And many thanks to Michael Edmonds for the heads-up on this paper.)

This is in fact true for anything to do with moving into the ‘universities of the future space (with or without MOOCs). Any strategy for online learning must surely consider resourcing: provision not only of the hardware, software, and facilities needed to properly deliver a ‘blended’ curriculum that may combine both face-to-face and on-line delivery, but also of the professional development needed to ensure that educators have the pedagogical knowledge and skills to deliver excellent learning experiences and outcomes in what for most of us is a novel environment. For there’s far more to offering a good on-line program than simply putting the usual materials up on a web page. A good blended learning (hybrid) system must be flexible, for example; it must suit

the interests and desires of students, who are able to choose how they attend lecture – from the comfort of their home, or face-to-face with their teachers. Additionally, … students [feel] the instructional technology [makes] the subject more interesting, and increase[s] their understanding, as well as encourag[ing] their participation… (Johnson et al., 2014).

This is something that is more likely to encourage the sort of critical thinking and deep learning approaches that we would all like to see in our students.

Furthermore, as part of that hybridisation, social media are increasingly likely to be used in learning experiences as well as for the more established patterns of social communication and entertainment (eg Twitter as a micro-blogging tool: Liyanagunawardena et al., 2013). In fact, ‘external’ communications (ie outside of learning management systems such as Moodle) are likely to become more significant as a means of supporting learner groups in this new environment – this is something I’m already seeing with the use of Facebook for class discussions and sharing of ideas and resources. Of course, this also places demands on educators:

Understanding how social media can be leveraged for social learning is a key skill for teachers, and teacher training programs are increasingly being expected to include this skill. Understanding how social media can be leveraged for social learning is a key skill for teachers, and teacher training programs are increasingly being expected to include this skill. (Johnson et al., 2014).

There is also a need, in any blended learning system, to ensure skilled moderation of forums and other forms of on-line engagement, along with policies to ensure privacy is maintained and bullying and other forms of unacceptable behaviour are avoided or nipped in the bud (Liyanawardena et al. 2013; Johnson et al., 2014). And of course there’s the issue of flipped classrooms, something that the use of these technologies really encourages but which very few teaching staff have any experience of.

Another issue examined by Liyanagunawardena and her colleagues, in their review of the MOOC literature, is that of digital ‘natives': are our students really able to use new learning technologies in the ways that we fondly imagine they can? This is a question that applies just as well to the hybrid learning model of ‘universities of the future’. In one recent study cited by the team, researchers found that of all the active participants in a particular MOOC, only one had never been involved in other such courses. This begs the question of “whether a learner has to learn how to learn” in the digital, on-line environment. (Certainly, I’ve found I need to show students how to download podcasts of lectures, something I’d naively believed that they would know how to do better than I!) In other words, any planning for blended delivery must allow for helping learners, as well as teachers, to become fluent in the new technologies on offer.

We live in interesting times.

And I would love to hear from any readers who have experience in this sort of learning environment.

T.R.Liyanagunawardena, A.A.Adams & S.A.Williams (2013) MOOCs: a systematic study of the published literature 2008-2012. The International Review of Research in Open and Distance Learning 14(3): 202-227

L.Johnson, S.Adams Becker, V.Estrada, & A.Freeman (2014) NMC HOrizon Report: 2014 Higher Education Edition. Austin, Texas. The New Media Consortium. ISBN 978-0-9897335-5-7

February 7, 2014

not science as I know it

This was first posted on my ‘other’ blog :)

By accident,  I came across the curriculum document for Accelerated Christian Education (ACE) which provides teaching & learning materials to parents who are homeschooling their children. New Zealand students who complete the program right  to year 13 gain university entrance.

Home Schooling NZ gives parents advice about the ACE program, but makes it clear that HSNZ does not work for Accelerated Christian Education or sell their teaching & assessment materials.  However, I was startled to see the following listed by HSNZ as one of the ‘distinctives’ [sic] of the ACE program:

Each student is taught from a biblical perspective developing critical thinking skills that will enable them to discern what is truly “…the good and acceptable and perfect will of God.” (Romans 12:2)

Having had a fair bit to do with the development of the Science section of the current national curriculum document, specifically, the Living World component, I was naturally interested in seeing how ACE handles a science curriculum. The answer is, poorly.

In fact, I feel that it’s most unfortunate that the ACE science program is officially recognised here, given statements such as this from Sir Peter Gluckman (the PM’s Chief Science Advisor) about the importance of science and science education. For example, from the curriculum overview material for grade 1 students we learn that students will

  • [pronounce and learn] new vocabulary words as they are defined and used in the text
  • [discover] God’s wisdom as he1 learns about God creating Earth…
  • [learn] about the design and care of the human eye and ear; high, low, soft and loud sounds.
  • [learn] about the importance of personal health – clean teeth and hands.
  • [gain] a respect for God as he learns about God’s wisdom, goodness, kindness, and that all things belong to God.
  • [read] stories and answer questions about God’s creation.
  • [continue] to build eye-hand coordination by drawing shapes, irregular shapes, and directional lines.

That’s it.

In contrast, the New Zealand Curriculum document has a number of subject-specific achievement aims for students at this level, in addition to those relating specifically to the nature of science. For example, students in their first year or two of primary school should

  • Learn about science as a knowledge system: the features of scientific knowledge and the processes by which it is developed; and learn about the ways in which the work of scientists interacts with society.
  • Appreciate that scientists ask questions about our world that lead to investigations and that open-mindedness is important because there may be more than one explanation.
  • Explore and act on issues and questions that link their science learning to their daily living.

Remember, that’s in addition to the achievement aims for biology (Living World), chemistry (Material World), earth sciences (Planet Earth & Beyond). and physics (Physical World).

And so it continues. I mean, how could this (from the ACE objectives for Grade 3) be construed as science by anyone assessing the document?

Studies Bible topics such as Jesus’ return; sin, death, and the curse; man’s freedom to choose to love and obey God.

Or this?

Discovers the Bible to be the final authority in scientific matters.

Science, it ain’t. It would appear that helping students to gain and enhance critical thinking skills isn’t on the curriculum either – after all, teaching students to look to authority for the answers runs completely counter to encouraging critical thinking and teaching students how to weigh up evidence.

While I haven’t read all the PACEs available for the curriculum, partly because I am not going to buy them in order to do so, I have read through the samples available on line. Among other things, the materials I viewed encouraged rote learning rather than deep, meaningful understanding of a subject – a long way indeed from current best-practice models of teaching & learning.

However, others have read ACE’s PACE documents, & have been extremely critical of them. The Times Education Supplement, for example, was startled to find that ACE materials available in 1995 contained the claim that the Loch Ness Monster has been reliably identified and seems to be a plesiosaur. (It seems this reference has since been removed from new textbooks published in Europe.)

The TES also addressed some rather trenchant comments to the UK educational body responsible for giving the ACE curriculum equivalent status to O and A level examinations. Perhaps the NZ equivalent of that body should give the ACE documents a closer second look.

 

1 No female pronouns used, that I could see. (No room for female scientists in this curriculum, either – students are introduced to ‘early men in science’.)

 

December 12, 2013

Evaluating teaching the hard-nosed numbers way

[This is a copy of a post on my blog PhysicsStop, sci.waikato.ac.nz/physicsstop, 10 December 2013]

Recently there’s been a bit of discussion in our Faculty on how to get a reliable evaluation of people’s teaching. The traditional approach is with the appraisal. At the end of each paper the students get to answer various questions on the teacher’s performance on a five-point Likert Scale (i.e. ‘Always’, ‘Usually’, ‘Sometimes’, ‘Seldom’, ‘Never’.)  For example: “The teacher made it clear what they expected of me.” The response ‘Always’ is given a score of 1, ‘Usually’ is given 2, down to ‘Never’ which is given a score of 5. An averaged response of the questions across students gives some measure of teaching success – ranging in theory from 1.0 (perfect) through to 5.0 (which we really, really don’t want to see happening).

We’ve also got a general question – “Overall, this teacher was effective”. This is also given a score on the same scale.

A question that’s been raised is: Does the “Overall, this teacher was effective” score correlate well with the average of the others?

I’ve been teaching for several years now, and have a whole heap of data to draw from. So, I’ve been analyzing it (for 2008 onwards), and, in the interests of transparency, I’m happy for people to see it.  For myself, the question of “does a single ‘overall’ question get a similar mark to the averaged response of the other questions?” is a clear yes. The graph below shows the two scores plotted against each other, for different papers that I have taught. For some papers I’ve had a perfect score – 1.0 by every student for every question. For a couple scores have been dismall (above 2 on average):

Capture1.JPG

What does this mean? That’s a good question. Maybe it’s simply that a single question is as good as a multitude of questions if all we are going to do is to take the average of something. More interesting is to look at each question in turn. The questions start with “the teacher…” and then carry on as in the chart below, which shows the responses I’ve had averaged over papers and years.
Capture2.JPG
Remember, low scores are good. And what does this tell me? Probably not much that I don’t already know. For example, anecdotally at any rate, the question “The teacher gave me helpful feedback” is a question for which many lecturers get their poorest scores (highest numbers). This may well be because students don’t realize they are getting feedback. I have colleagues who, when they give oral feedback, will prefix what they say with “I am now giving you feedback on how you have done” so that it’s recognized for what it is.
So, another question. How much have I improved in recent years? Surely I am a better teacher now than what I was in 2008. I really believe that I am. So my scores should be heading towards 1.  Well, um, maybe not. Here they are. There are two lines – the blue line is the response to the question ‘Overall, this teacher was effective’, averaged over all the papers I took in a given year; the red line is the average of the other questions, averaged over all the papers. The red line closely tracks the blue – this shows the same effect as seen on the first graph. The two correlate well.
Capture3.JPG
So what’s happening. I did something well around 2010 but since then it’s gone backwards (with a bit of a gain this year – though not all of this year’s data has been returned to me yet). There are a couple of comments to make. In 2010 I started on a Post Graduate Certificate of Tertiary Teaching. I put a lot of effort into this. There were a couple of major tasks that I did that were targeted at implementing and assessing a teaching intervention to improve student performance. I finished the PGCert in 2011. That seems to have helped with my scores, in 2010 at least. A quick peruse of my CV, however, will tell you that this came at the expense of research outputs. Not a lot of research was going on in my office or lab during that time.  And what happened in 2012? I had a period of study leave (hooray for research outputs!) followed immediately by a period of parental leave. Unfortunately, I had the same amount of teaching to do and that got squashed into the rest of the year. Same amount of material, less time to do it, poorer student opinions. It seems a logical explanation anyway.
Does all this say anything about whether I am an effective teacher? Can one use a single number to describe it? These are questions that are being considered. Does my data help anyone to answer these questions? You decide.

December 9, 2013

shaking up the academy? or how the academy could shake up teaching

Last week I spent a couple of days down in Wellington, attending the annual symposium for the Ako Aotearoa Academy. The Academy’s made up of the winners of the national Tertiary Teaching Excellence Awards, so there are around 150 or so of us now. While only 35 members were able to make it to this year’s event (& the executive committee will survey everyone to see if there’s a better time – having said that, everyone seems so busy that there’s probably no date that would suit everyone!), we had a great line-up of speakers & everyone left feeling inspired & energised. I’ll blog about several of those presentations, but thought I would start with one by Peter Coolbear, who’s the director of our parent body, Ako Aotearoa.

Peter began by pointing out that the Academy is potentially very influential – after all, it’s made up of tertiary teachers recognised at the national level for the quality of their teaching, & who foster excellence in learning & teaching at their own institutions.  But he argued – & I agree with him – that there is room for us to become involved in the wider scene. Peter had a number of suggestions for us to consider.

First up, there’s a lot going on in the area of policy – are there areas where the Academy might be expected to have & express an opinion? For example

  • There’s the latest draft of the Tertiary Education Strategy (TES), which “sets out the Government’s long-term strategic direction for tertiary education; and its current and medium-term priorities for tertiary education.” There’s a link to the Minister’s speech announcing the launch of the draft strategy here.
  • In addition, the State Services Commission’s document Better Public Services: results for New Zealanders sets out 10 targets across 5 areas. Targets 5 & 6 are relevant here as they are a reference point for government officials looking at evidence for success in the education sector. (Such scrutiny is likely to become more intense in light of the 2012 PISA results, which have just been made public.) Target 5 expects that we’ll “[increase] the proportion of 18-year-olds with NCEA level 2 or equivalent qualification”; #6 is looking for an increase in ” the proportion of 25 to 34-year-olds with advanced trade qualifications, diplomas and degrees (at level 4 or above)”. This will increase the pressure on institutions to increase retention & completion rates – might this have an effect on standards?
  • There’s also the requirement to achieve parity of success for ‘priority’ learners, especially Maori & Pasifika – this is priority #3 in the TES. (Kelly Pender, from Bay of Plenty Polytech, gave an inspirational presentation on how he weaves kaupapa Maori into pretty much everything he does in his classroom, in an earlier session.) And it’s an important one for us to consider. Peter cited data from the Ministry of Education’s website, ‘Education Counts’, which showed significantly lower completion rates for Maori & Pasifika students in their first degrees compared to European students, and commented that this will likely become a major issue for the universities in the near future.
  • If we’re to meet those achievement requirements, then how institutions scaffold learners into higher-level study, through foundation & transition programs, will become increasingly important. What are the best ways to achieve this?
  • Peter predicted increased accountability for the university sector (including governance reform). Cycle 5 of NZ’s Academic Audits has begun, and “is to be framed around academic activities related to teaching and learning and student support.” This is definitely one I’d expect Academy members to have an opinion on!
  • He also expects strengthened quality assurance processes throughout the education sector: this suggests a stronger (& more consistent) role for the  NZ Qualifications Authority, with the development of partnership dialogues across the sector (ie including universities).

Then, at the level of the providers (ie the educational institutions themselves – & that’s not just the polytechs & universities), we have:

  • a targeted review of qualifications offered at pre-degree level – there’s background information here;
  • a government-led drive to get more learners into the ‘STEM’ subjects (science, technology, engineering, & maths) – this poses some interesting challenges as, at university level, we’re seeing quite a few students who’ve not taken the right mix of subjects, at the right NCEA level, to go directly into some of the STEM papers they need for, say, an engineering degree;
  • the rise in Massive Open On-line Courses, or MOOCs. (I find these quite strange creatures as they are free to the student and typically attract very large enrolments, but also apparently have very low completion rates. What’s in them for the institution? A good way of offering ‘taster’ courses that hook students in?)
  • the likelihood that we will see the development of a system for professional accreditation of tertiary teachers (I’ve written about this previously and will write another post fairly soon, as accreditation was the subject of a thought-provoking session at the symposium);
  • how we achieve protection of academic standards – it’s possible that government policies (eg those linking funding to completion & retention rates) may result in a tendency to exclude of underprepared kids &/or lowering standards – neither is desirable but both are possible results of those policies.

That’s a big list and the Academy can’t do everything! So, what should it focus on? (This is not a rhetorical question – it would be great to get some discussion going.) The Academy, in the person of its members, is effectively a resource; a body of expertise – can it become a ‘go-to’ body for advice? Speaking personally I think we need to make that shift; otherwise we remain invisible outside our individual institutions & the teaching-focused activities we’re involved in, & in a politicised world that’s not a comfortable thing to be. Can we, for example, better promote the significance of teaching excellence outside the education sector? Become involved in the discussions around & development of any accreditation scheme? Develop position papers around maintaining teaching excellence in the context of the new TES?

What do you think? And what shall we, collectively, do about it?

December 5, 2013

nz’s pisa rankings slip, & the soul-searching begins

Filed under: education, science teaching — Tags: , , , , — alison @ 11:06 am

The latest PISA results are out, and NZ – despite remaining in the ‘above the average’ group for OECD countries – has nonetheless  slipped in this measure of achievement in reading, maths administered by the Programme for International Student Assessment . This is of concern, & there are probably multiple complex causes for our decline. Certainly the previous PISA commentary (2009) recommended that we pay attention to matters of inequality (There’s interesting commentary here, & also on the RNZ website.)

This morning’s Dominion-Post (I’m in Wellington at the moment, at a teaching symposium) carries a story giving a primary-teaching perspective.There are two key issues here: many primary teachers lack a science or maths background; and primary teachers in general are not well supported to teach these specialist sujects. (The removal of specialist science advisors - something I’ve commented on previously – did not help things.) This is important, because if students don’t gain a good understanding of these subjects – and good experiences of them! – during primary school, then they’ll basically be playing catch-up when they arrive in specialist secondary school classrooms.  Sir Peter Gluckman’s suggestion (in his report Looking ahead: science education in the 21st century) that each primary school have a ‘science champion’ would help here, but in the medium-to-long term it would probably be even better if intending primary school teachers received much greater exposure to the STEM subjects to begin with.

Should we worry? Yes, but I definitely agree with Fiona Ell, from the University of Auckland, who’s quoted in this morning’s Herald as saying:

People get very hung up on the ranking … because it’s like a Top of the Pops top 10 thing. I don’t think they should be ignored … but knee-jerk reactions to rankings are really dangerous in education systems.

So, there are issues that we need to address, and as Fiona’s pointed out, there are no quick fixes – we need to deal with them in a considered way that includes as many variables as possible (i.e. not just practices in schools).

One of those issues is highlighted by Sir Peter Gluckman, the Prime Minister’s Science Adviser, who’s said:

What’s worrying is that there seems to have been a decline in the people represented in the top end of the scale and an increase in the number of people at the bottom end of the scale.

And socioeconomic status may well play a part in this. From the Herald story:

New Zealand was one of just two countries in which socio-economic status had a strong connection to a student’s performance. Some countries’ education systems made up for social disadvantage, but this was not the case in New Zealand.

So any solution addressing the PISA results will of necessity be complex. It’s not going to be sufficient to look only at what’s going on in schools. Yes, support and professional development for STEM teaching across the compulsory sector will be needed. The quality of teaching is definitely important (for a student’s perspective see the Herald article). But without also seriously considering and attempting to deal with the social inequalities in this country, I suspect changes in the educational sector alone will not be enough.

November 21, 2013

learning leadership

and yes, that’s an intentionally ambiguous title :) (The full version was Learning leadership: the interplay between our own professional development and our classroom practice.)

I recently gave a pecha kucha** presentation on this subject, at an Ako Aotearoa mini-symposium up in Auckland. The idea for the subject of my presentation leapt to the front of my mind while I was at a Teaching Network*** meeting looking at how to raise the profile of teaching in tertiary institutions (specifically, universities). One of my colleagues kicked off part of the discussion with a brief talk on developing leadership in teaching, & I thought, all this applies to leading/guiding our students to become better learners, as well. Which is pretty much the thrust of my presentation. I pretty much use slides as talking points:

Slide2

 

Slide3

Slide4

Slide5

Slide6

Slide7

Slide8

Slide9

Slide10

And we finished up with some ideas on what future-focused leadership in teaching and learning could look like.

It would be really good to hear your thoughts on this :)

 

 

** a maximum of 20 slides, with a maximum of 20 seconds per slide. Certainly forces you to focus your ideas. Mine wasn’t that long, because I wanted to use it to spur discussion & so we need time for that in my short presentation slot.

*** an in-house group for staff from across the institution with an interest in all things to do with teaching.

Older Posts »

The WordPress Classic Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 164 other followers