The Outcome of Limiting Learning

Liz Morrish argues that learning outcomes are really not that important for either lecturers or students.

This post is about creating opportunity by freeing ourselves from some of our learning, teaching and assessment practices which set limits on the development of important graduate abilities like understanding that there are different forms of knowledge, different sources of knowledge, and that all knowledge does not originate from authority. Students need to integrate this awareness into the process of developing confidence, resilience and academic judgement. Indeed, an earlier post on the so-called ‘teaching REF’ argued that this is precisely the sort of learning gain which will be assessed. 

Educationalist Julian Stern (2013) tells us that the most essential characteristic of a successful educational experience is SURPRISE. Over the past ten years in UK higher education, we have been sucked deeper and deeper into the conformity exacted by regulatory bodies, but also of our own design. We are in danger of offering a rather repetitive, initiative-draining experience to students. I offer examples of two overly-restrictive practices: learning outcomes and formative assessment.

If we teach in universities, I’m sure we all do a decent job of drafting module specifications. We all know how to harmonize our objectives with subject and course objectives, cascading from benchmark statements and aligning with our university’s nominated graduate attributes. And then we put them in a folder and forget about them. I have yet to meet an academic who can tell me the learning objectives of any of their modules without consulting the specs. They can, of course, tell me all about the content, and what they hope the students will take from the module. Even transferable skills are more relevant to the learning and teaching process than learning objectives. So let’s free ourselves from that particular constraint.

Assessment has become more like a contract of sale than an educational experience. Anyone who has been in universities for a decade or more will recognise this scenario. You have a curriculum review, and there you discover the previous guidelines have all been reversed. In the last exercise I experienced, we did a quick 180 and overturned the framework which had considered multiple forms of assessment a virtue. No more assessment for learning, or assessment for equality and accessibility. Instead, we should confine ourselves fewer types of assessment, and fewer points of summative assessment. Intermediate feedback should be provided by formative feedback, AKA teaching to the test, and closely aligned with clear indicators of achievement against learning outcomes.

The danger I see in both learning outcomes and formative assessment is that we evacuate any possibility of surprise in learning. If it’s the 5th of November, it must be Semantics. Take the online practice test next week. But more damaging for the prospects of our students is that we have emptied our curriculum of any risk that they will need to exercise their own academic judgement. If I set a project brief for students, they expect me to advise on the research question, the nature of the data, the number of subjects, and the methods of analysis they should employ. This is the nature of formative feedback. I often feel I am assessing my own work. I recognise that students would benefit from pursuing group work, but even then, anxieties about dilution of individual marks compromises that learning experience, and makes it a hazardous venture for both students and staff. Formative feedback requires endless rehearsal for the summative event. Far from reducing anxiety, it makes assessment loom very large all through the module. What if I could assess my students on the basis of what they learn from risk taking, exercising judgement and initiative?

It is a long time ago since I taught in the USA, but I do remember that I had much more opportunity for flexibility of delivery and assessment. I could give them an unannounced quiz anytime I wanted. I could set an essay. I could walk in and announce I was cancelling the final exam. I could give them all As if they deserved them. If a visiting speaker was coming to talk on a relevant topic, I could assign the class to go, and expect a summary paper from each student. There was a syllabus, but I felt free to deviate from it. Alas, this Garden of Eden no longer exists, as the US has led the way in consumerist approaches to higher education.

So what if we had the flexibility to be responsive to current events? I was incredibly lucky one year, that former Australian Prime Minister Julia Gillard gave *that* speech on sexism and misogyny on the very day I was approaching the topic of sexist language in my module. What a gift, and what an inspiration to the class that day. But what if I had been scheduled to teach another topic? Learning opportunity missed. Learning is a wandering, progressive, transformative process. Yes, we need to impose clarity, structure and analysis, but let’s not make that the ultimate goal of all our modules, because we are missing some really important learning opportunities. Perhaps we should assess the strength of modules with just one question on course evaluations: “what surprises have you had in this module?”

Not Doing a Thomas Cook

The phrase “doing a Ratner” has its origin in the famous address given by Gerald Ratner, at the time, Chief Executive of the Ratners Group of jewellers, to the Institute of Directors, when he said this about his company’s product: “We also do cut-glass sherry decanters complete with six glasses on a silver-plated tray that your butler can serve you drinks on, all for £4.95. People say, ‘How can you sell this for such a low price?’ I say, ‘because it’s total crap'”.

Since that incident, companies have sharpened their sensitivities to reputational damage, either deliberate or inadvertent. This concern has only deepened with the growth of social media. In fact, absolutely the quickest way to get a reply from any customer service outlet is to call them out on Twitter.

This week, a sad and complex case came to a close. Thomas Cook travel agents had spent nine years avoiding taking responsibility for what a recent inquest has found to be a breach of their duty of care to a family whose children lost their lives due to a badly-maintained heating appliance in a holiday hotel in Crete in 2006. Finally, in May 2015, the chief executive took responsibility, but only after a large insurance payout to the firm had attracted further negative coverage in the media.

By contrast, after a recent Amtrak train derailment in the US (Tuesday 12th May) on the line from Philadelphia to New York in which eight people were killed, the president and CEO, Joe Boardman, sent all passengers (including me) an email on Saturday 16th May which included this:

“With truly heavy hearts, we mourn those who died. Their loss leaves holes in the lives of their families and communities. On behalf of the entire Amtrak family, I offer our sincere sympathies and prayers for them and their loved ones. Amtrak takes full responsibility and deeply apologizes for our role in this tragic event”.

If my confidence in Amtrak had ever wavered, this response restored it immediately. This is a company which recognizes failure, apologizes and intends to address the problems. Everything is acknowledged publicly. Bereaved families were not having their grief aggravated by being treated to secrecy and silence.

These are both great tragedies for all affected by them, and I do not wish to diminish them by making inappropriate comparisons.  However, universities could learn something from this. In the UK and the US, university managers increasingly view their institutions in terms of business and markets. No slur is allowed to attach itself to the university’s name. Social media policies, and behaviour codes on ‘civility’ have emerged, which in some cases threaten academic freedom and academic careers.

Civility initiatives have been defended by Nigel Thrift, vice-chancellor of Warwick University, in 2012 in the US Chronicle of Higher Education, so his reaction to apparent insubordination from Professor Thomas Docherty in 2014 was not surprising. Docherty’s offense was reported to be nothing more than sighing and making ironic comments in meetings. What the academic community suspected, was that his views on managerialism and audit culture in universities had attracted the rage of the university’s senior management. Professor Docherty spent an uncomfortable nine months suspended from his position, unable to use campus facilities or correspond with students and colleagues. All charges were later dismissed, and Professor Docherty returned to work. Like footballers mobbing a referee after a controversial decision, Warwick management’s strategy was not intended to change a result, but to exact more cautious behaviour from him and others in future. Beware the next time you appear to be critical of your university, or even the state of higher education, especially if you cannot draw on prominent academic standing, and support from the Times Higher, among many others. But reputational damage has been done to the Warwick ‘brand’; many in the academic community now question their commitment to academic freedom, and freedom of speech.

This mistrust was heightened when, in July 2014,  a lawyer in the firm acting for the university, SGH Martineau, posted a blog piece which seemed to validate macho management techniques ‘pour encourager les autres’.  Titled ‘getting your teeth into high performer misconduct’, it was offensive for many reasons, not least because it made an explicit analogy between the conduct of star footballer Luis Suarez, who had recently bitten an opponent during a game, and academics who ‘damage their employer’s brand’ by their outspoken opinions. It is worth quoting from the piece in full:

“Universities and colleges may, equally, encounter high performing employees who, although academically brilliant, have the potential to damage their employer’s brand. This could be through outspoken opinions (where these fall outside the lawful exercise of academic freedom or freedom of speech more widely) or general insubordination, e.g. a failure to comply with the reasonable requests of an employer, or other behaviour such as bullying or harassment of colleagues. Irrespective of how potentially valuable these employees may be to their institutions, the reality is that, in consistently accepting unacceptable behaviour, institutions may be setting dangerous precedents to other employees that such conduct will be accommodated. From a risk perspective, it is also much harder to justify a dismissal, or other sanction, if similar conduct has gone unpunished before”. []

I may be mistaken, but I recall that the piece was edited, and the clause in brackets added, after a Twitter flame fest broke out between an outraged academic community, and SGH Martineau. The amendments just cemented the impression that academic freedom was merely an inconvenient afterthought.

Another current controversy drawing press attention is the climate of sexual harassment on many UK and US university campuses. In the US, President Obama and Vice President Biden have made personal commitments to ending the rape culture on campus. Even in the face of presidential support, it is still not uncommon for universities to try and block attempts to research the issue, as recently happened at the University of Oregon.

Here in Britain, ‘laddism’ is seen as part of a campus culture which emerges in a climate of hyper-masculinity, sports teams and alcohol use (Phipps and Young 2015). It is almost always pack behaviour and characterised by sexism, misogyny and homophobia. There is no doubt that it forms a hostile environment for women students. A 2010 report funded by the National Union of Students, Hidden Marks, found that one in seven women had experienced sexual assault, and 68% had been verbally harassed. The National Union of Students conducted another survey in 2015 which found one in three students had been sexually assaulted or abused while at university

This issue has made the news consistently in the intervening five years, however, a report which featured on the front page of The Guardian on 25th May 2015 revealed that fewer than half of the Russell Group universities systematically record rapes, sexual assaults or sexual harassment.

Despite this accumulation of evidence, Dr Wendy Piatt, the director general of the Russell Group, is quoted in the article as saying: “Russell Group universities take the issue of any kind of harassment, abuse or violence against women extremely seriously indeed. Our institutions have robust policies and procedures in place to deal with these matters, because ensuring student safety and wellbeing is extremely important to us.” []

It might be more accurate to say that universities have robust procedures to make sure that instances of sexual assault and harassment are kept well buried. Even bystander intervention campaigns can make senior managers nervous in case their university is seen as a ‘rape campus.’ It is ironic, given the emphasis we are all supposed to place on the much-vaunted ‘student experience,’ that this is subordinated to reputational concerns whenever there is a conflict.

Universities are on morally shaky ground in persisting with a stance of denial and negligence, but increasingly they are legally vulnerable as well. In the US, some women students are filing discrimination suits under Title IX legislation (the 1972 law that prohibits sex discrimination at colleges that receive US federal funds) claiming that universities do not provide them with a safe learning environment. Students at Xavier University of Ohio and  Harvard University Law School are the most recent to have successfully pursued claims to force universities to institute training and response to sexual assault.

In my own work as a lecturer in linguistics, I read projects and dissertations every year that draw on recordings of actual conversations among students. I am only too aware of male students’ discourses of rape and sexism, as well as racism. This year, I have learned a lot from being part of a ‘respect and consent’ group at my university as we move to create culture change and a better environment for all students, bolstered by an explicit code of behaviour and campaign of education. All UK universities would do well to take the issue seriously and follow the example of those universities who take women’s safety seriously.

As a general point, I hope university managers are learning something about brand management that they won’t find in business school leadership courses. Universities are charged with defending academic freedom, freedom to challenge authority, and the responsibility to provide a safe environment. The cases I have outlined have shown that an over-concern with reputation is, paradoxically, more than likely to lead to damaging your reputation. The lesson is – do an Amtrak, not a Thomas Cook.


McVeigh, Karen. 2015 Top universities fail to record sexual violence against women. The Guardian. Accessed May 24th 2015.

NUS report: That’s what she said: women students’ experience of ‘lad culture’ in higher education. 2015.

Phipps, Alison and Young Isabel. 2015. Lad culture in higher education: agency in the sexualisation debates. Sexualities. ISSN 1363-4607

It’s Metricide – Don’t Do It

Demand for universities to release more and more data from their students is growing. Thus far, it has focussed on the academic quality of teaching and research. A new departure is to measure universities on the salaries of their graduates. Liz Morrish gives the background.

If you thought that feeding the audit vultures with data for “better performance metrics in higher education” would placate them, you would be wrong. The last blog discussed a small section of the Conservative manifesto (see last blog post 23rd May). Let’s return to it, because there is still an unexamined proposition – to “require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”. Once again, the focus is on graduate outcomes, but this time they are entirely financial. This truly sinks the last nail in the coffin of higher education as intellectual self-fulfilment. The manifesto anticipates a little-publicized outcome of the recently passed Small Business, Enterprise and Employment Bill 2015 (SBEE) 

Here’s a quick summary of the purpose and intent of this new legislation. First, to put it in context, let’s take a step back to the 2010 Browne Review which recommended a tripling of university fees in England. Lord Browne, the Labour government which commissioned the review, and the coalition government which acted on it, never imagined that ALL universities would charge the £9000 maximum fee. They assumed they had tweaked the right drivers and incentives to ensure that fees would mirror university reputation and thus perceived ‘value for money’. Instead of forming an orderly hierarchy, universities all moved to charge the maximum, or near to it, for fear of signalling an inferior ‘product’. This has landed the government with a far bigger outlay on student loans that they ever intended or budgeted for. There is now a very large hole in the balance sheet for the Department of Business Innovation and Skills, and to make matters worse, the RAB charge (Resource Accounting and Budgeting) seems to grow with each new estimate (the RAB charge is the portion of student borrowing that will not be paid back). This may be in excess of 45% of the sum borrowed, and is a debt which BIS will need to service.

And so, having vastly inflated the actual public spend on higher education, albeit through the agency of student borrowers, BIS and the government need to find a way to make this improvident model sustainable. Taking an idea from President Obama, one way to reclaim the money is by ensuring that graduate salaries exceed the threshold for repayment. This is no easy deal in the current economic climate, and so the government’s ‘nudge’ unit must have been employing all their imagination towards their solution – the FEER. It stands for the Future Earnings and Employment Record. In an era of big data, it has become possible to link the following records to individuals: university attended (and possibly even subject studied), amount owed in tuition and maintenance loans, and, via HMRC tax records, the amount that a graduate currently earns. This intrusive leakage is now permissible since the passing of the SBEE. BIS can simply ask for these records in order to compile them into a league table of graduate loan repayments, by university. What better way to weaponize that data than to try and influence student choices, cast as ‘aspirations’ in the legislative text, or even punish universities for having the temerity to confer degrees on deadbeats who cannot repay their loans?

You know you’re in trouble when the discourse turns to ‘journeys’ and ‘destinations’, but it gets worse. Although the government’s Education Evaluation fact sheet constitutes a total failure of logic, it displays a discursive masterstroke,  with a chaining of ‘learning outcomes’, ‘performance data’, ‘accountability’, ‘interventions’, and then serving the whole salad up as a solution to ‘social mobility’. And the final section re-designates universities as mere factories for the production of labour inputs: “This data, presented in context, will distinguish universities that are delivering durable labour market outcomes and a strong enterprise ethos for their students”.

So, that is the future. Applicants for university courses will be invited to consider Key Information Sets including projected earnings, and make their choices accordingly. More worryingly, will universities fear the FEER to the extent they will discriminate against women – seriously at risk of defaulting on loans with all those inconveniently timed maternity leaves? Will universities continue to offer course that lead to lower-paying graduate jobs: nursing, teaching, fine arts? Who knows what knowledge and skills may be in demand in ten years’ time? Having sacked off excellent chemistry, physics, zoology and sociology departments, universities are full of forensic science, criminology and equestrian studies courses. These are all popular ‘vocational’ subjects, but lead to mixed outcomes in terms of employability and earnings. Meanwhile, graphic artists and English graduates seem to be climbing the salary scales with recent developments in computer games.

We can only hope that students are not as mercenary as their political masters. Students, I imagine, will continue to make choices based on love for their chosen subject, desire to remain in their home city, or to move to a new one, to attend a university to be with their current partner, or the one that gives them the opportunity to study overseas – or any of the countless other factors that motivate student choices. I may be institutionalized in an arts and humanities faculty, but I have met very few students in any university I have visited who had graduate salary as top of their aspirations. But, then, the government thinks I live in an ivory tower, insulated from economic realities. Nevertheless, I’d bet my perception of students is more accurate than either their initial estimate of the RAB charge or their strategy for retrieving it.

My Loss is Your Learning Gain

Liz Morrish discusses some new ways the Conservative government will seek to assess and rank universities. ‘Learning gain’ is about to be ‘a thing’.


It is just over two weeks after the General Election, and our thoughts turn to the prospect of more cuts in public spending, a new leader for the Labour Party, some uncertainty over Brexit and the referendum on EU membership, and, post UKIP, a somewhat muted dialogue over immigration. But what lies in the future for higher education? Have you been paying selective attention over the months leading up to the election? A tuition fee cut may have lodged in your memory, but that was Labour Party policy, and we can forget that now. What does a Conservative government have planned for universities? We know that abolition of the cap on student numbers was already in the offing, as was a national postgraduate loan system for taught masters and PhD courses. That is not the interesting bit. The game-changer is encoded in this section:

“We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality… and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”.

Value for money: As students pay higher fees, they have been demanding more classroom time, better feedback and higher-grade campus facilities. These have largely been delivered. The focus is now shifting to what the government recognises as value for money – are students learning anything worthwhile? For the last twenty years, as universities have come under greater pressure to justify (especially non-vocational) courses, we have been speaking the language of generic or ‘transferable’ skills. At my university, we make the claim that graduates will be able to demonstrate attributes such as information, organisational and communication skills, and also ‘intellectual agility’ defined as, “aptitude for independent, critical thought and rational inquiry, alongside the capacity for analysis and problem-solving in multiple contexts” [NTU Strategic Plan 2010-2015 p. 10]. Now, it looks as if we will have to bolster that claim with actual hard evidence. All in the name of accountability – a call which never fails to extend its dominion.

A framework to recognise universities offering the highest teaching quality: This is what has been flagged as the Teaching REF. The good news is, at last we have found a use for learning outcomes. It was probably inevitable that as soon as we were obliged to position a degree course as merely an accumulation of learning outcomes, and higher education as a set of generic skills, that one day, this would be called to account. Even better news, demonstrating “the highest teaching quality” is likely to shine a spotlight on students, not on academics, who usually occupy the panopticon. The generation of students who have become used to earning reputational credit for their primary and secondary schools via SATS tests, may now be called forth in the service of their universities. And so, step forward the concept of ‘learning gain’. Simply put, can our graduates demonstrate the nominated transferable skills to a greater extent than those without the benefit of higher education?

There has already been progress made towards measurement of these generic skills, and two parties engaged in this discussion are the OECD (Organisation for Economic Co-operation and Development) and HEFCE (Higher Education Funding Council for England).

The OECD’s AHELO (Assessment of Higher Education Learning Outcomes) has been piloted. The long-term intention is to assess subject learning outcomes, but so far, this has only been attempted for engineering and economics. Generic skills are better candidates for assessment, not least because there already exists an ‘instrument’ to do this: the Collegiate Learning Assessment (CLA). This test aims to “to evaluate the critical-thinking and written-communication skills of college students. It measures analysis and problem-solving, scientific and quantitative reasoning, critical reading and evaluation, and critiquing argument, in addition to writing mechanics and effectiveness.” It uses scenario-based problems and requires students to marshal evidence, and then evaluate the risks or merits of particular solutions or options. It is assumed that this will reflect on “instructional effectiveness” and enable a better understanding of “the teaching-learning interplay”.


Whether or not we will adopt this particular testing mechanism in the UK, as a result of the Tories’ demand for accountability, is not yet clear. The CLA is one of the options being considered, but HEFCE is also currently consulting on the concept of ‘learning gain’.

“We wish to build better ways of capturing excellent educational outcomes, including new approaches that measure students’ learning gain, and of refining existing indicators of students’ learning experiences and progression to employment or further study”. []

The executive summary of the OECD report details some caveats. It mentions a risk that AHELO data could be used as yet another a ranking tool, and emphasises that this was never the intention. I think, though, we can write the script on this one if (when) ‘learning gain’ assessment goes ahead. It will, of course, be employed in a culture inflected by neoliberal politics which requires markets, competition and hierarchies. It can only end one way. And so we can anticipate the second fear listed in the report – that the assessment will become the basis for (re)allocating resources, either from research to teaching, or from apparently unsuccessful universities to those with more ‘intellectually agile’ students. The only silver lining to this cloud is that I suspect many of the most intellectually agile, creative students may be found outside the more favoured universities.


academic argument

No sooner has the largesse of REF2014 been promised than invitations to the next UK academic Olympiad are being issued. The prospect of REF2020 has unleashed a new university obsession with rankings and evaluations of research. In many departments, researchers are undergoing a process of continual audit. Every grant application is vetted; every publication is read by internal, and perhaps external, assessors. In the current system, grades are assigned 1-4, but everyone knows that the stakes have been raised and only grades 3 and 4 count. Let’s take a step back and ponder why this might not be such a good idea. Any such strategy of continuous assessment of research is bound to be inaccurate. Here are some reasons why:

  • We do not know what rules are going to be applied in 2020, but they are sure to be different from 2014. Who foresaw the insertion of the ‘impact’ monkey-wrench at the last minute? Similarly, we cannot assume the current weighting given to publications in the next REF formula.
  • Published work has already been assessed by the most appropriate scholars in the field during the process of peer review. In its ideal form, this is what we describe to students as ‘formative assessment’.  What added value could a summative assessment from a generalist offer?  It is ironic at a time when we have attempted to curb the effect of high-stakes assessments on our students, in the realisation that such exercises are of limited value in ‘the real world’, that we now impose those same stressors on ourselves.
  • Any internal ranking is likely to be distorted either by relations of power within a department, or by personal animosities. Can a Reader offer a fearless appraisal of the work of a Dean? Can a colleague overlooked for promotion dispassionately review the work of someone whose success they envy?
  • I think we all now know that there were some very strange GPAs and rankings emerging from REF2014. Biological sciences appear to have won the laurels, and over half the units submitted achieved a GPA of 3 or above. The sociology panel, meanwhile, delivered a harsher and more partisan verdict. Those units which reflected government priorities in their impact studies scored well; the more ‘critical’ areas of the subject were slammed. Internationally recognised departments and research groups can now legitimately be targeted for closure by their university’s management. And so a vindictive government has made vice-chancellors their willing proxies, simply by persuading them to swap evaluation of research quality for evaluation by £££££.
  • Because of 4), rankings of units, and therefore of publications, is likely to follow one of two courses: scores will either reflect a privileging of government/ managerial bias against critical scholarship, or, every single subject will look at the celebrity status of biological sciences and conspire to inflate their scores in order to attain parity of esteem.

Ranking of publications is also undesirable because of the distorting effect it will have on academic work and relationships:

  • It is hard enough for some scholars to submit their work to peer review and public scrutiny in the first place. If we have to contemplate the scorn of a head of department, or senior colleague – why bother? I imagine a lot of people will allow other agendas – teaching enhancement, student experience, employability to occupy them instead.
  • If senior researchers – likely to be the most ‘productive’ scholars – are the ones doing all the assessing and ranking of pieces, how will they have time to pursue their own work? Inevitably, this will eat up much of their research time, and also a great deal of their ‘leisure’ time. The research capacity of many departments will begin to ebb away under the strain of constant audit.
  • People will become secretive about publication plans, or leave publication until the last possible moment before the next REF. They may perseverate on the minimum 4 publications, and allow other ideas to dissipate unfulfilled and unexplored. This is unhealthy for the future of university research.
  • We risk creating departments of people who despise and distrust each other. How long will Dr A hold a grudge against Professor B for a Grade 2 assessment? Will payback come when Dr A is asked to observe Professor B’s teaching?

So, it is very clear that the value of assessing work either pre- or during REF is unreliable. It is retained simply for its disciplinary effect, and because management think they can summon up research quality like a genie from a lamp. The analogy is pertinent, because they seem to imagine that a ranking score equals quality-made-visible, and that this is some pathway to quality enhancement. It will be the reverse, of course, especially when we consider the amount of research time and funding bound to be sacrificed in the process. And on top of this loss, we will create unhappy, dysfunctional departments where every colleague nurtures resentment. UK academics have seen their job satisfaction and self-esteem falling low enough. Let’s not compound the issue with another ill-founded exercise. This is hysteria masquerading as rationality. Just make it stop.

Critical university studies, discourse and managerialism