Metrics Metrics Everywhere and not a drop to drink

Yeah, this isn’t going to be a rant about the types of metrics that ‘engaged’ me somewhat at a meeting yesterday. Rather, it’s about the metrics we use in setting assessment that seem to cause fairly significant damage to student learning: word counts and hours of effort.

Firstly let’s establish one thing: there are occasions where strict adherence to a word count is an authentic twist in an assessment that mirrors many real scenarios. I have no contest with those, and generally have no contest with the idea of helping students tailor their effort to the task in hand. There are few real scenarios when a bit of paper dictates how many hours you should spend doing something. Our modules are typically 15-credit and as such come with a few metrics: 150 hours of student effort, 4000 – 5000 words if you assess the whole lot in one essay, and no more than 9 intended learning outcomes and preferably fewer. The modular system is often maligned for creating chunks of content and a perception that once you’ve passed one module you can forget it all when you move onto the next. The benefits of the modular system are rarely spoken of: it’s convenient to pace content and make assessment seem manageable over fixed time frames.

Word limits have somehow mutated from a useful measure of how much is sufficient and a prompt to write concisely in order to show off knowledge and research, into a target. I’m sure we all have anecdotes of students who are just aiming to get to the word count and with little consideration for the quality of the work within that. I’ve had students convert numbers to words, remove hyphens and various other tricks to get to that magic word count minus 10% threshold lest they should incur a penalty. Conversely, word counts do level the playing field a bit – no extra credit for substantially more effort and more words. I can’t dispute that one. Unfortunately there comes a time where just seeing the word count total creep up past the target is not a guarantee of a half-way decent mark. Personally I tend to aim for about 25% under a wordcount so I can edit and refine the ideas, adding additional detail and having a good cull of waffle (that’s what blogging is for).

Recommended hours are almost immediately contradicted by staff. For our project modules we have a recommendation of how many ‘lab hours’ must be spent. Lab hours include prep and analysis, as well as time spent actually doing reactions. We contradict ourselves straight away because we know it will vary between projects. Some research projects are simply more labour intensive than others. We deal with this somewhat by having greater expectations for the lighter projects: if all of your reactions take 10 days of heating in a furnace, you’d be expected to do a lot more reactions than a student who must purify everything via chromatography or other time consuming methods. We also expect a strong degree of multi-tasking (particularly at M-level) where students can have a reaction on, work up a reaction and be planning a reaction. The usual comments about checking spectroscopic data before proceeding apply. Recommended hours start out looking like a good idea, and do have some merits in ensuring fairness. Like word counts, we do not want students getting additional credit for being sufficiently privileged as to be able to work very long hours without concerns over health, caring responsibilities or part-time work.

Time management is an important skill to develop and research projects offer a great opportunity to do this. I’m starting to regret not asking students to produce some kind of detailed plan with timescales for their projects. It would be unfair of me to do so when it is not required of other students. But recommended hours are open to abuse in many ways. Another clash is when the expectations of the supervisor does not match the abilities of the student. It’s quite easy for those with good lab skills to say ‘my student should be able to make these 10 compounds’ and then be critical of them when, in the recommended hours, they cannot do so. And sometimes that is due to errors – if a student can put in a few extra hours to overcome the obvious disadvantage of dropping their flask in the rotary evaporator bath, that may be a good thing. But putting in tens of additional hours just to meet expectations is probably not.

Overall word counts and recommended hours are useful to a point. I’ve not really touched on recommended hours for other types of assessments such as exams. I have opinions there but none that I can fully articulate at this time. I would urge students to focus more on the quality of the time they spend on the research (and be honest about time spent doing versus time spent gossiping or being distracted in the lab), and on the quality of the words on the page rather than their number.