Category Archives: metrics

‘low quality’ higher education…again

This blog post appeared on the Wonkhe site in mid-November 2020. Written by Nick Holland, Competition and Registration Manager at the Office for Students, it had been anticipated by a statement by Universities UK the previous day. In this article, UUK clearly aligned themselves with the presuppositions that there is a serious problem with quality in UKHE and that the sector needs to introduce some proper regulating. This plays into widely-held beliefs amplified by some media outlets hostile to universities, and a government which seeks to undermine them along with other pillars of a democratic society like the legal apparatus, a free press and even parliament itself.

It is dispiriting for those of us who have had quite a lot of faith in the UK’s HE regulatory mechanisms from the Quality Assurance Agency to the internal procedures of validation and periodic review that operate within all universities. Less able to command confidence has been the government’s own regime of the Teaching Excellence Framework, operated by the Office for Students.

Nevertheless, Nick Holland, the author of the piece, appears innocent of these established structures. We apparently exist in some post-lapsarian quality vacuum which needs to be pumped with new regulations, about which OfS has launched a consultation. And so OfS promises to act on new proposals, even if it requires a temporary adjournment of its other preoccupations: freedom of speech  and grade inflation – another piece by Nick Holland.

So below, with some commentary, are excerpts from Holland’s blog. What his choice of discourse does is to install a set of presuppositions about the endemic poor quality of higher education. Furthermore, in this depiction, no evidence can be relied upon apart from a limited set of proxy measures which lend metric infallibility to the conferral of quality. This intervention from the Office for Students seems to reverse several years of avowed ‘light-touch’ regulation in which oversight enabled universities to operate their own bureaucracy of quality assurance and enhancement. But now, the principal metric to be trusted to certify the worthiness of higher education is that of graduate salaries. This is the fulcrum which has elevated the concept of higher education as private good while depressing the notion of higher education as public good. 

*****

Excerpts from Holland in grey and interpretation in green

While we had planned to consult on our approach to quality and standards in any case, we will of course draw on our experience of regulating through the pandemic in our future regulation of quality and standards.

This was conducted through veiled threats demeaning the quality of online provision with no thought about how we might intervene to support universities. That’s just not what we do.

The OfS has always been able to hold universities and other higher education providers to account for the quality of their courses and the standard of qualifications they award. But these proposals would sharpen our regulatory requirements, raise expectations for quality and student outcomes, and allow us to take action where there are poor quality courses at providers in particular subject areas.

We already have the regulatory framework well embedded in practice and working well. But since the bar for moral panics has gone way north in 2020, we thought we’d create a bit of spontaneous drama.

Quality and standards

Also included are secure standards – so students can be assured that their degree will stand the test of time – and successful outcomes. Subject to consideration of responses to the consultation, we would be looking to use these definitions as part of our regulation – underpinning the baselines we set which all providers must meet in order to be, and remain, registered with us.

Of course, the enduring worth of your degree would be best ensured if we were committed to ensuring that your HEI continues to function. Instead, we are committed to ‘the market’ and providing registration to new, ‘challenger’ institutions which are awarded degree awarding powers almost immediately. It’s hard to imagine a better way to undermine the hard-won accreditation of your university. And what do we mean by ‘outcomes’? Well, that changes all the time. Sometimes it means ‘good degrees’ but then we clamp down when universities give too many of them. We’re a bit clearer what it means this month later on.

Crucially, we are also saying that all higher education providers must provide quality for all groups of students. That means two things. First, if we are worried that certain groups of students are being adversely affected, we can swiftly intervene. Second, we are saying – unequivocally – that is not acceptable for providers to use the proportion of students from disadvantaged backgrounds they have as an excuse for poor outcomes. Doing so creates a skewed playing field, where poor performance for disadvantaged students is effectively allowed. That is untenable, and unfair to those students who have often overcome the odds to enter higher education. Obstacles to attainment for these groups need to be removed and not hidden behind.

See what I did there? First I install a presupposition that not all HEIs provide a quality education for all students. That charges universities with the responsibility to fully compensate for the social, educational, cultural and systemic inequalities that particular groups of students encounter. For example, the long periods of missed education that disabled students or care-experienced students have faced. And don’t make excuses for the alienation that many BEM or gay students have experienced all through school – that’s up to universities to fix. It’s helpful to scapegoat universities as it makes us popular with government when we undermine public trust in them.

The plans also allow us to intervene at a subject level if we have concerns. This intervention really matters. Most universities and other higher education providers offer high quality higher education across the board.

I had to say this. But important to rattle that sabre every time I throw them a compliment.

But at some providers, we have been concerned about pockets of low-quality provision. Being able to intervene at subject level will make a real difference. As well as assessing data on student outcomes we will also continue to welcome notifications from students alerting us to issues and concerns about the quality of their course.

Obviously, you’d be better making your views known through the various structures available at your HEI, from course committees, to student unions or even the NSS. But here at OfS, while we don’t endorse ‘cancel culture,’ we do approve of snitch culture, and we have always envied the direct reporting line offered by ‘rate your professor’.

Toughening up

Choosing a section title that’s maximally offensive especially during a pandemic.

Ensuring students have every opportunity to achieve successful outcomes on their courses remains an important OfS priority. That is why we are focusing on the number of students who progress to the end of their course and go on to managerial and professional employment or higher-level study. We are proposing to update – and toughen – our requirements for the minimum performance we would expect from any university or other higher education provider.

I told you we’d be clearer what ‘outcomes’ means. In 2020, we’ve pretty much settled on it meaning getting a well-paid job. Because we want students to pay back their loans. Yes, we’ve reduced the whole university experience to pretty much that.

Deciding on the numerical values for these minimum baselines will take time and be subject to further consultation, but we will have higher expectations for providers for all of their students.

I think we can guess what that baseline will be…because we want students to pay back their loans.

These are important proposals, which will help to properly protect students. Please do take the time to have your say in the consultation, which runs from today until January 12.

Don’t waste your time. We’ve already decided.

*****

That subject level data again…..

It’s fascinating to trace the meandering semantics and pursuit of ‘low quality degrees’ over the years. It has meant many things, from accusations of ‘lamentable’ teaching to complaints that too often degrees do not lead to high-paying jobs. While the government boldly alleges poor quality of provision and outcomes in the public HE sector, by contrast, they are happy to support a favoured ‘challenger institution’ by conferring (via QAA) degree awarding powers on the Dyson Institute of Engineering and Technology before it has even had a graduating cohort.

Such indulgence is not bestowed on other institutions and an assemblage of metrics must be found to identify courses that the government rather wishes would disappear. Chief among them, we suspect, might be media studies, gender studies, creative arts and the more critical humanities and social science subjects generally.

David Kernohan of WONKHE (30th November) writes that OfS were due to publish data and a report entitled Start to Success representing the latest attempt to devise a diagnostic quality tool. To save them the trouble, Kernohan has produced a draft version mapping data points representing all HE courses in the UK against projections of non-continuation of students on courses and data from Graduate Outcomes on “highly skilled” graduate employment. Kernohan himself suspects that this metric may be low quality.

What we find is that many of the courses the government denounces, or that universities have been busy eradicating, perform well against these metrics. In the lower quadrant of the quality mapping are quite a lot of law, business, civil engineering and computer science courses, while in the higher quadrant are many courses in Arabic, Classics, English and Geography. Also, some courses at higher status universities manifest surprisingly low scores, while some less-favoured courses like media studies at lower-ranked universities turn up scores above 9/10. Like Kernohan, I am not attempting to confer credibility on these metrics; instead we need to recognise that in searching for ‘low-quality courses’ to liquidate in the name of regulatory scrupulosity, the casualties may not be the government’s preferred candidates. If the government wants to purge those courses which develop critical questioning and an awareness of social justice, then coercion and apparent metric infallibility will not deliver that result. They will have to emulate the more authoritarian approach adopted by Victor Orban in Hungary. They will need to name the courses they disapprove of and ban them. I wonder how close we are to that eventuality? The culture warriors are laying the groundwork and a receptive post-truth society might see it through.

TEF metrics: proxy, toxic and idiotic…but very neoliberal

This is an updated version of a previous blog post and apeared recently on the LSE Impact blog. It also draws on my article, The Accident of Accessibility: How the Data of the TEF Creates Neoliberal Subjects, published in Social Epistemology and appeared in a special issue on Neoliberalism, Technocracy and Higher Education; Guest Edited by Justin Cruickshank and Ross Abbinnett.

The stated aim of the Teaching Excellence Framework (TEF) is to encourage excellence in teaching in higher education and to provide information for students to make improved decisions about the courses they take at university. In this post, I argue that contrary to these goals, the TEF is only marginally interested in teaching quality and instead contributes to the increasingly personalised and transactional nature of university education, provides perverse incentives for educators and ultimately positions participants in higher education as neoliberal subjects.

The stress occasioned by constant demands for academics to submit to evaluation has been well documented (Loveday 2018Morrish 2019), nevertheless the bureaucratic appetite for surveillance is fed by the ease of accessing quantifiable data. This blog looks at the repercussions for universities, students and academics of the chosen metrics of the Teaching Excellence Framework outlined below.

In 2016, the government published a White Paper on plans to reform higher education, Success as a Knowledge Economy (SKE). The brainchild of Jo Johnson, former minister for universities, it laid out a justification for the introduction of the TEF as the solution to a series of imagined faults with universities, such as ‘lamentable’ teaching, and a lack of ‘return on investment’ for graduates of some university courses. Johnson promised that it would never become ‘big, bossy and bureaucratic‘.

The selection of metrics for the TEF appeared arbitrary: student satisfaction with teaching, assessment and feedback (taken from the National Student Survey, NSS), continuation (retention) rates (taken from the Higher Education Statistics Agency, HESA) and a measurement known as Longitudinal Educational Outcomes data (LEO). These proxy measures have nothing to do with classroom teaching, and yet have become imbued with spurious validity.

David Beer writes that metrics measure us in new and powerful ways such that they order the social world and shape our lives. Whereas, metrics can appear neutral and necessary, in the case of TEF proxy metrics, we can argue that their whole purpose is dirigiste. Since the introduction of tuition fees in 1999, successive governments of all hues have sought to reconfigure universities as instruments of market ideology.

Each of the TEF proxy metrics underpins this strategy: the NSS arose from the project to transform students into consumers; the retention rate metric was similarly designed to measure consumer appeal, even though it reflects more accurately the social advantage of the student body; LEO data became available with the passing of the Small Business, Enterprise and Employment Act. This legislation permits mining of individual tax records for integration with student loan and university degree records and enables the government to assess which courses produce graduates most able to repay their tuition fee loans. It becomes apparent, in the context of some of Jo Johnson’s remarks (20152016a2016b), that the role of the TEF was to shift student choice towards STEM subjects which, on average, lead to higher-paying careers, which the government judges to be more beneficial to the economy.

These metrics are not simply benign and have serious implications for academics whose labour is being evaluated along new and unforeseen dimensions. For instance, they must now privilege the potential popularity of a subject before proposing to teach it. They must justify both new and current courses on the grounds of ‘employability’. And so, innovation in teaching and research is determined by appeals to economic value and the capricious choices of 18-year olds, rather than by the advance of knowledge or professional judgement of academics. Nevertheless, universities which fail to score highly on the TEF metrics risk being deemed ‘failing’ and the government may seek to pressure them to close courses which do not deliver the right ‘outcomes’. These market reforms were skillfully packaged as enhancing student ‘choice’. Furthermore, market ideology and tenets of neoliberalism such as competition, choice and individual responsibility are all bolstered discursively throughout the SKE White paper and by repetition by politicians.

A neoliberal discourse 

Significantly, the government avoids the word ‘university’, insisting that that there has been insufficient penetration of ‘the market’ in the higher education sector, and so in order to instil ‘competition’ and ‘choice’, the market must be opened up to new providersProvider also implies a transactional function for universities – a kind of cash for credentials scenario in which the institution simply provides its product to its ‘customers’ in exchange for money.

Investment, and its cousin, return on investment (ROI), are further linked discursively with teaching quality, in a way that implies that receiving information about the latter would in turn ensure a guarantee of the former: ​“The quality of teaching should be among the key drivers of a prospective student’s investment” ​(SKE Ch 2 para 8 p43). However, evidence reveals that there is little material correlation. Reports by the Institute for Fiscal Studies analyse the government’s Longitudinal Earnings Outcome data (Bellfield and Britton 2018Britton et al. 2016) and show the outcomes-require-competition assumption to be a myth. The best predictors for graduate earnings (the likely meaning of investment here) is found to be parental income and prior attainment. Return on investment in this way has arguably been used deliberately to reposition higher education as a private, rather than a public good.

Although the government has claimed the TEF will empower students, the Office for Students,  which was supposed to  ensure that student choice is placed at the centre of university policy making, fell silent when a student voice was raised in challenge. In 2017, a great deal of manufactured outrage was expressed in right-wing newspapers when a University of Cambridge student, Lola Olufemi, expressed a request for the English department to more fully decolonise the curriculum. Nor has recent higher education legislation or market discipline enabled a broadening of student choice when we consider that, since the implementation of the Browne Review funding system of tuition fees, there has been a precipitous decline in the availability of part-time routes for undergraduate study.

Conclusion

The accident of accessibility of particular metrics, inasmuch as they overlap with neoliberal priorities, has therefore determined which data will serve as Key Performance Indicators in the TEF. What may seem to be a random assortment of proxy data points, has in fact  served an agenda to refashion universities, staff and students as neoliberal subjects. The TEF audit therefore only appears to be tangentially concerned with quality. Instead it incentivizes universities towards a more pronounced concentration of business and science curricular provision and is reinforced via discourse which all participants in the framework are compelled to cite.

There is a discussion page for articles in the special issue of Social Epistemology which readers can find here.

Pressure Vessels: The epidemic of poor mental health among higher education staff

This blog originally appeared on the HEPI website: and accompanies HEPI Occasional Paper 20. I am grateful to HEPI and its director, Nick Hillman, for the opportunity to produce this report.

There has recently been a significant amount of media concern surrounding the poor mental health of academics. See also.

In February 2018, Paul Gorczynski of the University of Portsmouth claimed that more academics and students have mental health problems than ever before, with findings that 43% of academic staff exhibited symptoms of at least a mild mental disorder. This is nearly twice the prevalence of mental disorders in the general population.

The degree of overwork is confirmed by a work-life balance survey published by Times Higher Education, as is the extent to which this damages mental health. Students also recognize that being taught by exhausted and depressed staff impairs their experience of university courses.

A report from Nottingham Trent University UCU confirms that workloads, for many academics, have become unmanageable. In the NTU UCU survey, the majority of staff surveyed reported working at least one unpaid half day at weekends, with 18% working a whole day. 41% felt that their workload had had a negative effect on their mental health. In fact only 6% said that their workload did not impair their mental health. Typical symptoms included disrupted sleep, depression and cognitive impairment.

Pressure Vessels: The epidemic of poor mental health among higher education staff, is an Occasional Paper which sets out the scale of the problem based on a Freedom of Information survey showing a rising rate of staff referrals to counselling and occupational health in universities. The report also examines the factors which academics have identified as key causes of stress.

New workload models are at the top of the list, especially those which do not reflect the actual time necessary for completion of a task. Typically, for teaching assignments, one classroom hour will be accompanied by an additional 1.6 or 1.7 academic related hours on the annual workload. Lecturers are now required to provide teaching materials across several platforms. Each lecture will now need a Powerpoint, or similar, presentation, associated handouts and, perhaps, an interactive study guide. The lecture video will require editing and uploading. Student email enquiries will need responses. You will need to supply marking criteria which reflect your assessed learning outcomes. For all assessed tasks, you must upload an exemplar response, with commentary. All of these will, of course, need to be redone if you make any changes at all. And, since about 50% of lecturers are on precarious or casual visiting contracts, more of them will be writing those lectures for the first time. Technology may have transformed academics’ ways of working but has not necessarily lightened their load. Nevertheless, academics are often obliged to collude in the fiction that all of this, together with marking an entire cohort’s assessments, can be discharged in 1.7 hours. No wonder academics are stressed.

Many people outside academia imagine it is a life of contemplation and long summer holidays. In fact, few academics are able to take all of their annual leave allowance, and the NTU UCU study shows that 30% of academics worked on at least 10 days during their annual leave. The Times Higher Education 2018 work-life balance survey quoted one academic “We are not allowed to book holidays during term time, but we are also not allowed to book holidays over exam periods, marking periods, internal exam board meetings or course development meetings. This means that although we get 35 days [of holiday] a year, we actually only have opportunity to use about 14.” Senior members of staff find that non-teaching weeks during the summer are now taken up with reading and assessing the research outputs of their department colleagues for submission to the Research Excellence Framework audit. Junior academics are asked to staff weekend open days for potential applicants to undergraduate and postgraduate degrees. And let’s consider as well that the majority of academics will be under pressure to produce published research at high quality and volume. The extent of this work, too, fails to be reflected in staff workloads. Responsibilities like editing academic journals, peer reviewing articles and book manuscripts, examining theses, giving expert testimony – many of these are unforeseen at the start of any academic year and therefore mount up, but remain invisible to the human resources manager.

Academics are also evaluated by an assortment of research metrics: citation counts, the impact factor of the journal in which it is published, and the amount of research grant money obtained. These are all poor proxy measures of research quality, but they are easy to track. Despite their obvious limitations, academics are forced to accept that metrics have become the currency of performance management in universities. To work there means giving yourself over to forensic surveillance, and also being willing to have your closest friends and colleagues scrutinize your work, in both teaching and research. That pressure is cumulative and to many, the university has become an ‘anxiety machine’.

As one academic on Twitter out it, academics have been “forced by management into stress positions”. There have, to date, been two deaths by suicide linked to conditions of work at UK universities. Cardiff University lecturer, Malcolm Anderson, took his life on campus when, after years of appeals to his manager, his workload had escalated out of control. Another death by suicide was triggered by pressures to meet targets for research funding. Professor Stefan Grimm at Imperial College, London, had been rebuked by his manager over this issue.

I am honoured that Professor Mike Thomas, former Vice-Chancellor of the University of Central Lancashire, has written a foreword. He has conducted research into the area of compassion and kindness in leadership. Universities need to adopt these values because no profession can continue to drive its employees to burnout and alienate its pool of graduate recruits. There are remedies: sustainable career pathways, more autonomy and control over workload and responsible use of metrics. But beyond this, the simple, humane values of kindness and care for each other must be recovered if we are to ameliorate the toxic university.