the carnegie hall hypothesis: practice makes perfect

There’s another paper out on ways to improve student performance in university science papers, this time with a focus on biology. Again, what follows is a cross-post from something originally written for the Bioblog.

Hot on the heels of the paper on methods for improving learning in first-year physics (Deslauriers, Schelew & Wieman, 2011), comes one by Haak, HilleRisLambers, Pitre & Freeman (2011) that casts a critical eye on methods for teaching first-year biology classes.

Today’s students come from more diverse backgrounds, and have far more diverse prior learning experiences, than when I was a student myself. Those differences can contribute to a gap in achievement in first-year biology – something that’s exacerbated by academic assumptions about prior learning & which can contribute to poor student retention into subsequent study in the subject. 

Interestingly, when I showed the paper to a couple of colleagues, their first response was ‘let’s get clickers!’. Personally I’m rather cool on the idea: partly because they’re not exactly cheap, but also because what both papers show is that it’s not the clickers themselves that make the difference, it’s what you do with them. If all you do is use them to find out what answers students give to multichoice questions, & nothing more, this technology won’t actually add anything to student learning (eg Deslauriers et al., 2011). If, however, clickers are used as an integral part of a wider, active learning, experience, then you’ll see biiig improvements in student learning outcomes.

And that is amply demonstrated by this latest study by Haak & his colleagues. They note that, especially when dealing with disadvantaged students in the US, the response has often been to throw money at the problem to support reasonably comprehensive heavily targeted programs. Because this can quickly become rather expensive, such programs rarely become a regular, normal part of teaching programs. And they ask:

Can an existing STEM course [1] be modified to improve performance by students from disadvantaged educational and socioeconomic backgrounds who are at high risk of failing, without requiring increased resources in the way of staffing or external funding?

In other words, is it possible to set things up in a large-group lecture classroom that lets students achieve as they would if they were getting one-on-one instruction? This is something that should be of interest to university academics for several reasons, including the fact that in New Zealand there is an increasing focus from the government on improving student retention and completion rates, and on enhancing the proportion of Maori & Pasifika students enrolling & succeeding in university programs.

To answer this question, the research team worked with a big first-year Biology class at the University of Washington, specifically looking at the performance of students in the university’s Educational Opportunity Program (EOP). Students in this program come from disadvantaged backgrounds (educationally &/or economically), & the majority of them are from non-Caucasian ethnic groups. Analysis of a very large number of student records found a large ‘achievement gap’ between EOP and non-EOP students, such that over the period 2003-2008 EOP students in Biology 180 had an average failure rate of 21.9% (cf 10.1% for the non-EOP cohort). Haak & his colleagues hypothesised that this was because grades in the paper were heavily dependent on exams that ‘test higher-order cognitive skills’, and that students in the EOP program aren’t as well prepared as the non-EOP group to that assessment style.

The paper was originally taught by the standard, traditional lecture format, with little involvement by the students. Previous work led by one of the team (Freeman) found that if the lecturer incorporated active-learning exercises in his class (daily multichoice questions and weekly practice tests), then all students’ performance improved compared to the outcome from that traditional format. This makes sense, as the students were practicing the skills they’d need to achieve well in the final exam.

But wait, there’s more. A third course design saw the class taught (by the same instructor) without any lectures at all, where the active-learning exercises were combined with ‘pre-class reading quizzes and extensive informal group work in class’ – exactly what Deslauriers & his team did with their experimental cohort of physics students.  No surprises here:

The highly structured [third] approach resulted in another increase in overall performance by all students, compared with the low-structure, lecture-intensive course with no required active learning and [my emphasis] the moderate structure design based on clickers and a weekly practice exam.

That in itself is an excellent outcome. What about the EOP students in particular, since that’s where the big achievement gap is apparent?

… although all students benefit from [highly-structured teaching], EOP students experience a disproportionate benefit.

Way to go! Importantly, in these straitened economic times, this intervention didn’t cost any extra money. What’s more, the second time the ‘highly structured’ intervention was used, class size had gone from 345 to 700, lab clases had been cut to one every 2 weeks, and the ratio of teaching assistants to students went from 1:49 to 1:87.5. (Note to the Finance people: this is not a reason to cut funding for demonstrators!)

You could ask how, exactly, this intervention is having its effect. Are the students simply learning more ’stuff’ as a result of the different teaching methods, or are they also gaining higher-order cognitive skills? During Cathy Buntting’s PhD research she found that teaching students how to develop concept maps had a significant impact on their ability to answer ‘thinking’ questions, as opposed to ‘recall’ questions, so I’d have put money on Haak’s team finding that active learning has a positive impact on cognitive abilities. Haak & his colleagues comment that because Biology 180 relies heavily on higher-order thinking-type questions in its exams, then better results in those exams does suggest ‘actual learning gains’ and an improved understanding of the content covered in the paper. They suggest that

active learning that promotes peer interaction makes students articulate their logic and consider other points of view when solving problems, leading to learning gains.

Hopefully this will be the focus of a future research project.

Deslauriers L, Schelew E, & Wieman C (2011). Improved learning in a large-enrollment physics class. Science (New York, N.Y.), 332 (6031), 862-4 PMID: 21566198

Haak DC, HilleRisLambers J, Pitre E, & Freeman S (2011). Increased structure and active learning reduce the achievement gap in introductory biology. Science (New York, N.Y.), 332 (6034), 1213-6 PMID: 21636776

[1] STEM = Science, Technology, Engineering & Mathematics

Oh, and the ‘Carnegie Hall’ hypothesis? It’s named for the story of a tourist who asked a New Yorker how to get to Carnegie Hall. The local guy answered, ‘practice!’

engaging students effectively in science, technology and engineering

This is another little something that I originally wrote for the Bioblog. It’s a look at a new report published by Ako Aotearoa, the organisation charged with promoting and enhancing tertiary teaching excellence here in New Zealand.

My eye was caught by that title to a paper just out on the Ako Aotearoa website (click here for the summary document & here for the full report). The sub-title is The pathway from secondary to university education, a topic that is dear to my heart.

Tim Parkinson & his co-authors were keen to get a handle on just how university students make the transition from secondary school to university, and how they become/remain engaged with science during that process. The project’s underlying aims were to:

• improve student engagement in the study of science at university;
• improve the transition from the school learning environment to that of university;
• identify and promolgate pedagogical ‘best practice’ for science education in the first year at university.

(I know this is nit-picking, but surely the aim was to provide information that will help universities enhance student engagement and transition, using a range of ‘best practice’ options identified during the project. They weren’t looking at whether particular interventions actually had that result.)

In order to know how to make these changes, you really need to know what’s currently happening – and also how lecturers & students percieve what’s happening in their classrooms. We already know (eg Buntting, 2006) that there’s a mismatch between lecturer & student perceptions about prior knowledge, in biology at least, so I think it’s a fairly safe bet that the same mismatch exists around perceptions of teaching quality and engagement. The research team looked at all this using a combination of questionnaires & focus groups, working with secondary school science students (N=421), university students in their first year of a science degree (N=630), school science teachers (N-33) and uni science lecturers (N=69). Each of the four groups in the study answered the same questions, although the wording differed a bit depending on the group. For example,

Teacher questionnaire: I give students the opportunity to influence the way that they are taught. Student questionnaire: I am given the opportunity to influence the way I am taught.

(Parkinson et al, 2011; answers were scored on a 5-point Likert scale.)

As you might expect, it turns out that lecturers’ style, personality & enthusiasm had a big impact on students’ engagement with science at university, and on their ability to move smoothly from secondary school to higher-level study. But the lecturers’ abiltiy to present information in contexts that students see as relevant to their own specific interests is also important – not least because this would allow students to fit that information into their own internalised understanding of & knowledge about science (their ‘schema’). In addition

learning science in a contemporary context… stimulates engagement, and students enjoy learning when it is connected with a sense of discovery.

And there were definitely notable differences in perceptions related to teaching and learning. For example, the team commented that

… school and university students thought less highly of the abilities of their teacher in [the area of teacher qualities ie things like presentation skills, quality of feedback] than did the teachers and lecturers themselves. For example, university and school learners perceived their lecturers’ qualities to be of a moderate standard, whereas lecturers themselves reported that their own lecturing qualities were of a high standard.

Something that I found intriguing was that none of the groups felt that self-directed learning was a significant facet of classroom activity – its reported frequency fell around ‘sometimes’ and ‘rarely’. Our graduate profile document indicates that we expect students to be independent learners by the time they complete their degree – developing the necessary skills must surely begin in first year! Surely there’s a need – noted by the researchers in their summary, to make sure that we reward such things as critical thinking and other higher-order learning skills (which of course has an impact on how we assess our students’ learning).

It is tricky for uni staff though, for our students come into class with a wide range of previous learning experiences, depending on what subjects and which standards they’ve studied at school. This means that we’re a bit between a rock & a hard place, needing to extend able students with a lot of existing content knowledge without losing those who might not have the same skills or learning experiences. Parkinson & his colleagues suggest that universities – certainly university staff engaged in first-year teaching – need to become much more aware of the learning outcomes gained by students in their NCEA studies. This would mean that those lecturers would be able to

build on the diversity of knowledge that results from the standards-based NCEA high school education.

It occurs to me that doing this would send a powerful message to students – that their lecturers really do care about helping manage the transition from school to uni and are personally interested in their learning outcomes. (I don’t mean to suggest that we aren’t, only that students may not perceive things that way!) And that can have a big impact on how students perceive and approach their studies.

C.Buntting (2006) Educational issues in tertiary introductory biology. PhD thesis, University of Waikato.

T.J.Parkinson, H.Hughes, D.H.Gardner, G.T.Suddaby, M.Gilling & B.R.MacIntyre (2011) Engaging students effectively in science, technology and engineering (full report) Ako Aotearoa ISBN 978-0-473-18900-6 (online)

effects of changing teaching styles on student learning

This is a repost of an item I’ve just written for my ‘other’ blog. It would be good to hear what others think of the teaching methods it examines :-)

I know I’m creeping into Marcus’s territory here but the research I’m going to discuss today would apply to pretty much any tertiary classroom :-)

This story got a bit of press about a month ago, with the Herald carrying a story under the headline: It’s not teacher, but method that matters. The news article went on to say that “students who had to engage interactively using the TV remote-like devices [aka ‘clickers’] scored about twice as high on a test compared to those who heard the normal lecture.” However, as I suspected (being familiar with Carl Wieman’s work), there was a lot more to this intervention than using a bit of technology to ‘vote’ on quiz answers :-)

The methods traditionally used to teach at university (ie classes where the lecturer lectures & the students take notes) have been around for a very long time & they work for some – after all, people of my generation were taught that way at uni, & it’s not uncommon to hear statements like, we succeeded & today’s students can do it too. But transmission methods of teaching don’t reach a lot of students particularly well, nor do they really engage students with the subject as well as they might. (And goodness knows, we need to engage students with science!)

Wieman has already documented the impact (or lack of it) of traditional teaching methods on student learning in physics, but this paper (Deslauriers, Schelew & Wieman, 2011) goes further in examining the effect on student learning and engagement of changing teaching methods in one group of first-year students in a large undergraduate physics class. It can be hard to manage a class of 850 students, and so the lecturers at the University of British Columbia had split it into 3 groups, with each group taught by a different lecturer. While the lecturers prepared and taught the course material independently, exams, assignments and lab work were the same for all students.

Two of the three groups of students were involved in the week-long experiment; one continued to be taught by its regular, highly experienced instructor, while the other group was taught by a graduate student (Deslauriers) who’d been trained in ‘active learning’ techniques known to be effective in enhancing student learning. And ‘active learning’ wasn’t just using clickers: the ‘experimental’ group had: “pre-class reading assignments, pre-class reading quizzes [on-line, true/false quizzes based on that reading], in-class clicker questions…, small-group active learning tasks, and targeted in-class instructor feedback” (Deslauriers et al, 2011). Students worked on challenging questions and learned to practice scientific reasoning skills to solve problems, all with frequent feedback from the instructor. There was no formal lecturing at all; the pre-class reading was intended to cover the factual content normally delivered in class time. While the control group’s lecturer also used clickers, this was simply to gain class answers to quiz questions & wasn’t used along with student-student discussion, which was the case with the experimental class.

One reason often given by lecturers for not trying new things in the classroom is that the students might resist the changes. But you can avoid that. I know Marcus finds his students are very accepting of change if he explains in advance what he’s doing & how the innovation will hopefully enhance their learning, and Deslauriers, Schelew & Wieman did the same, explaining to students “why the material was being taught this way and how research showed that this approach would increase their learning.”

So, what was the effect of this classroom innovation? Well, it was assessed in several ways.

During the experiment, observers assessed how much the students seemed to be engaged in & involved with the learning process; they also counted heads to see what attendance was like. At the end of the intervention, learning was assessed using a multichoice test written by both instructors – prior to this, all learning materials were provided to both groups of students. And students were asked to complete a questionnaire looking at their attitudes to the intervention.

In both classes, only 55-57% of students actually attended class, prior to the experiment. Attendance remained at this level in the control group, but it shot up to 75% during the experimental teaching sessions. Engagement prior to the intervention was the same in both groups, 45%, but nearly doubled to 85% in the experimental cohort. Test scores taken in the week before the experiment were identical for the two groups (an average mark of 47%, which doesn’t sound very flash) – but the post-intervention test told a completely different story. The average score for the control group was 41% and for the experimental class it was 74% (with a standard deviation in each case of 13%). And the intervention was very well-received by students, with 77% feeling that they’d have learned more if the entire first-year course had been taught using interactive methods, rather than just that one week’s intervention.

Which is fairly compelling evidence that there really are better ways of teaching than the standard ‘transmission-of-knowledge’ lecture format. I try to use a lot of interactive techniques anyway – but reading this paper has cemented my intention to try something completely different next year, giving readings before a class on excretion (a subject which a large proportion of the class always seem to struggle with), and using the lecture time for questions, discussion, and probably a quiz that carries a small amount of credit, based on the readings they’ll have done. And of course, carefully explaining to the students about what I’m doing.

I’ll keep you posted :-)

Deslauriers L, Schelew E, & Wieman C (2011). Improved learning in a large-enrollment physics class. Science (New York, N.Y.), 332 (6031), 862-4 PMID: 21566198