TEF metrics: proxy, toxic and idiotic…but very neoliberal

This is an updated version of a previous blog post and apeared recently on the LSE Impact blog. It also draws on my article, The Accident of Accessibility: How the Data of the TEF Creates Neoliberal Subjects, published in Social Epistemology and appeared in a special issue on Neoliberalism, Technocracy and Higher Education; Guest Edited by Justin Cruickshank and Ross Abbinnett.

The stated aim of the Teaching Excellence Framework (TEF) is to encourage excellence in teaching in higher education and to provide information for students to make improved decisions about the courses they take at university. In this post, I argue that contrary to these goals, the TEF is only marginally interested in teaching quality and instead contributes to the increasingly personalised and transactional nature of university education, provides perverse incentives for educators and ultimately positions participants in higher education as neoliberal subjects.

The stress occasioned by constant demands for academics to submit to evaluation has been well documented (Loveday 2018; Morrish 2019), nevertheless the bureaucratic appetite for surveillance is fed by the ease of accessing quantifiable data. This blog looks at the repercussions for universities, students and academics of the chosen metrics of the Teaching Excellence Framework outlined below.

In 2016, the government published a White Paper on plans to reform higher education, Success as a Knowledge Economy (SKE). The brainchild of Jo Johnson, former minister for universities, it laid out a justification for the introduction of the TEF as the solution to a series of imagined faults with universities, such as ‘lamentable’ teaching, and a lack of ‘return on investment’ for graduates of some university courses. Johnson promised that it would never become ‘big, bossy and bureaucratic‘.

The selection of metrics for the TEF appeared arbitrary: student satisfaction with teaching, assessment and feedback (taken from the National Student Survey, NSS), continuation (retention) rates (taken from the Higher Education Statistics Agency, HESA) and a measurement known as Longitudinal Educational Outcomes data (LEO). These proxy measures have nothing to do with classroom teaching, and yet have become imbued with spurious validity.

David Beer writes that metrics measure us in new and powerful ways such that they order the social world and shape our lives. Whereas, metrics can appear neutral and necessary, in the case of TEF proxy metrics, we can argue that their whole purpose is dirigiste. Since the introduction of tuition fees in 1999, successive governments of all hues have sought to reconfigure universities as instruments of market ideology.

Each of the TEF proxy metrics underpins this strategy: the NSS arose from the project to transform students into consumers; the retention rate metric was similarly designed to measure consumer appeal, even though it reflects more accurately the social advantage of the student body; LEO data became available with the passing of the Small Business, Enterprise and Employment Act. This legislation permits mining of individual tax records for integration with student loan and university degree records and enables the government to assess which courses produce graduates most able to repay their tuition fee loans. It becomes apparent, in the context of some of Jo Johnson’s remarks (2015; 2016a; 2016b), that the role of the TEF was to shift student choice towards STEM subjects which, on average, lead to higher-paying careers, which the government judges to be more beneficial to the economy.

These metrics are not simply benign and have serious implications for academics whose labour is being evaluated along new and unforeseen dimensions. For instance, they must now privilege the potential popularity of a subject before proposing to teach it. They must justify both new and current courses on the grounds of ‘employability’. And so, innovation in teaching and research is determined by appeals to economic value and the capricious choices of 18-year olds, rather than by the advance of knowledge or professional judgement of academics. Nevertheless, universities which fail to score highly on the TEF metrics risk being deemed ‘failing’ and the government may seek to pressure them to close courses which do not deliver the right ‘outcomes’. These market reforms were skillfully packaged as enhancing student ‘choice’. Furthermore, market ideology and tenets of neoliberalism such as competition, choice and individual responsibility are all bolstered discursively throughout the SKE White paper and by repetition by politicians.

A neoliberal discourse 

Significantly, the government avoids the word ‘university’, insisting that that there has been insufficient penetration of ‘the market’ in the higher education sector, and so in order to instil ‘competition’ and ‘choice’, the market must be opened up to new providers. Provider also implies a transactional function for universities – a kind of cash for credentials scenario in which the institution simply provides its product to its ‘customers’ in exchange for money.

Investment, and its cousin, return on investment (ROI), are further linked discursively with teaching quality, in a way that implies that receiving information about the latter would in turn ensure a guarantee of the former: ​“The quality of teaching should be among the key drivers of a prospective student’s investment” ​(SKE Ch 2 para 8 p43). However, evidence reveals that there is little material correlation. Reports by the Institute for Fiscal Studies analyse the government’s Longitudinal Earnings Outcome data (Bellfield and Britton 2018; Britton et al. 2016) and show the outcomes-require-competition assumption to be a myth. The best predictors for graduate earnings (the likely meaning of investment here) is found to be parental income and prior attainment. Return on investment in this way has arguably been used deliberately to reposition higher education as a private, rather than a public good.

Although the government has claimed the TEF will empower students, the Office for Students,  which was supposed to  ensure that student choice is placed at the centre of university policy making, fell silent when a student voice was raised in challenge. In 2017, a great deal of manufactured outrage was expressed in right-wing newspapers when a University of Cambridge student, Lola Olufemi, expressed a request for the English department to more fully decolonise the curriculum. Nor has recent higher education legislation or market discipline enabled a broadening of student choice when we consider that, since the implementation of the Browne Review funding system of tuition fees, there has been a precipitous decline in the availability of part-time routes for undergraduate study.

Conclusion

The accident of accessibility of particular metrics, inasmuch as they overlap with neoliberal priorities, has therefore determined which data will serve as Key Performance Indicators in the TEF. What may seem to be a random assortment of proxy data points, has in fact  served an agenda to refashion universities, staff and students as neoliberal subjects. The TEF audit therefore only appears to be tangentially concerned with quality. Instead it incentivizes universities towards a more pronounced concentration of business and science curricular provision and is reinforced via discourse which all participants in the framework are compelled to cite.

There is a discussion page for articles in the special issue of Social Epistemology which readers can find here.

Advertisement

The Office for Students: Ten reasons why it is not for students at all

The Office for Students (OfS) is the new regulatory body for universities and higher education providers in the UK.  To date it has had a short and rather volatile history. Below is a collection of the main issues which students and academics should be aware of.

1. The OfS will ensure that the Teaching Excellence Framework (TEF) becomes even more prominent for universities who are to be assessed on their ‘outcomes’. However, the TEF is relatively untested, and its critics charge that it will not diagnose poor teaching any more than it will uncover excellent teaching. It is not designed for these tasks since no teaching is actually observed. Teaching quality is inferred from proxy measures which have a very distant and disputed relationship with teaching. See this blog from Dorothy Bishop, and this previous one from me.
2. The TEF will not incentivise universities to prioritise teaching. Unlike the REF (Research Excellence Framework) which has, arguably, recognised and rewarded excellence in research wherever it is found (notwithstanding Derek Sayer’s well-founded objections), a very different set of circumstances obtain for the TEF. Let’s take an example. Several universities have seen fit to cut courses in Modern Languages in response to falling student demand. Languages other than English and Irish Gaelic will soon no longer be taught in Northern Ireland, so how would an undisputed finding of excellent teaching affect that decision? Will universities channel funding to support excellent teaching wherever it is found? I predict they will not, and that is because funding follows the student. It is a formula designed to disrupt the traditional right of universities to make autonomous decisions about course provision based on the current state of knowledge and discovery. The fact is, when university curricular decisions are outsourced to the caprice of 18 year olds, there is little point in trying to pretend any other factor counts. If you have decided to expand a course because it attracts funding and international students, then no amount of poor National Student Survey scores will not dislodge that conclusion.
3. Ergo, poor teaching will be condoned and concealed by universities in the flawed and distorted market of UK higher education. The TEF is still useful to universities as it offers a justification for getting rid of unconventional academics who are disliked by managers.
4. The Office for Students seems to fixate on issues which don’t really register as important for students. Amatey Doku, NUS Vice President for Higher Education, answered questions from The Joint Committee on Human Rights – a Select Committee of both Houses of Parliament on 17^th January 2018. Here he exposes the mythology of a crisis of freedom of speech in universities which is not top of students’ priorities.
5. The Office for Students has no representative from the National Union of Students on the board. This is in spite of promises from Theresa May that the NUS would work in consultation with the new regulatory body. The sole student representative, Ruth Carlson, is relatively unknown. The circumstances of her appointment are not clear, but the new minister for higher education, Sam Gyimah, revealed that she was chosen from outside of the pool of three candidates considered appointable by the interview committee. We can only speculate what advantages Ms Carlson’s appointment might confer on the board of the OfS, but expertise in student representation does not appear to be among them. She is studying civil engineering, however, and this might plug a gap on the board (see 6).
6. Not a single other scientist or engineer has been selected for the board.
7. The Office for Students’ mission is defined in Chapter 2 para 37 of Success as a Knowledge Economy, the government White Paper published in May 2016.

“The OfS will be explicitly pro-competition and pro-student choice, and will make sure that a high quality higher education experience is available for students from all backgrounds. For the first time, we will put the interests of the student at the heart of our regulatory landscape. By enabling better student outcomes, we will also protect the interests of taxpayers and the economy”.

But the suspicion at this point is that the government’s understanding of competition and choice is restricted to the introduction of new private providers into the system. The fear is that they will choose to provide cheap-to-teach courses, like law and business, and this will further restrict the choices available to students. This concern is grounded in the fact that among the members of the board are Carl Lygo, former VC of BPP University, part of the Apollo Group which includes the for-profit University of Phoenix in the USA. The rest of the appointees can be seen here  and we note that private sector and business professionals predominate over practitioners in higher education.

8. There are real doubts about how the quality of higher education courses will be protected by the new regulator. The OfS will oversee the award of university title to new HE providers – a privilege currently only bestowed by the Privy Council. The OfS has already shown signs that it may tolerate a less rigorous pathway to university status than we see with current arrangements. Alarm bells rang for many academics when the UA92 Manchester United Academy was announced. The new regulatory arrangements allow for degree awarding powers to be issued with no demand for a track record of quality teaching and assessment under the supervision of an established university.  OfS will also be able to revoke the title of university for those institutions it deems to be failing. The current quality assurance system works with universities if they are seen to be in need of improvement, but students now might start studying at a university, only to find their institution downgraded or fined into bankruptcy.

9. The OfS has already demonstrated poor judgement in its attempt to appoint Toby Young to the board. Given the structures outlined in the White Paper, this appointment must have been overseen by ministers (namely Jo Johnson), and Young would have been interviewed by Sir Michael Barber, the Chair of OfS. The appointment of student representative, Ruth Carlson (see point 5 above) seems similarly unorthodox. This action has alienated most parts of the sector, as we can only assume it was meant to. We need an independent regulator which can work with universities, not antagonise them for the sake of it.

10. Jo Johnson, the previous minister for higher education, has suggested that it will be within the remit of OfS to issue financial penalties to universities which award ‘too many’ firsts and 2.1 degrees. Firstly, as I argue (in a forthcoming piece), there is no firm basis for charging universities with grade inflation. Secondly, there is no suggestion at the moment what might constitute ‘too many’. If the OfS does interfere with universities’ cherished independence and academic judgement in this manner, it is unlikely to make many friends among students it counts as its central constituency.

The unease which has greeted the launch of the OfS has prompted Alistair Jarvis, Chief Executive of Universities UK, the vice chancellors’ representative body, to write of the recent consultation document from the OfS, “The tone of the document is, in places, confrontational and appears preoccupied by short-term political concerns rather than the larger long-term task of creating a credible, independent regulator”.

The OfS has shown itself to be willing to pursue moral panics that vice chancellors feel originate with a government piqued by perceived opposition to its agenda (especially Brexit).  Many of the rest of us resent the ideologically motivated campaign in both government and media circles which is unsympathetic to dearly held academic values such as education for the public good and worry that the OfS is merely another vehicle by which to instigate this. I for one share Alistair Hudson’s hope that, “In the months ahead, it will be necessary for the OfS to establish itself as a mature, fair and accountable regulator that uses its powers to support students through proportionate regulation and judgement.” Sadly, the shortcomings exposed by its initial actions have meant that OfS has probably exhausted any goodwill it might otherwise have been able to claim.

The accident of accessibility: How the data of the TEF creates neoliberal subjects

This is the link to a video of a talk I gave to the Digital University in a Neoliberal Age Symposium, organised by the Contemporary Philosophy of Technology Research Group. Title: The accident of accessibility: How the data of the TEF creates neoliberal subjects.

Abstract:

In an era of neoliberal reforms, academics in UK universities have become increasingly enmeshed in audit, particularly of research ‘outputs’ via the Research Excellence Framework (REF). A new Teaching Excellent Framework (TEF) has emerged in 2017, whose results are determined primarily by proxy data of National Student Survey (NSS) scores, retention data and Longitudinal Educational Outcomes data (LEO), i.e. salaries of graduates. This has been made possible by SBEE (Small Business, Enterprise and Employment Act 2015) legislation which has enabled data mining and synthesis of data streams from records held by the Student Loans Company (SLC), HMRC and universities themselves.

These two audit processes, REF and TEF, were originally envisaged as instruments to evaluate research and teaching, respectively, at institutional level. This had a distinctively neoliberal purpose in seeking to mould universities more closely towards serving the economic needs of the nation. The REF, however, has also been recruited as an instrument of individual performance management in universities, with each academic forced to compete in academic output and research funds with the most talented and unencumbered scholars. The TEF, similarly, bestows an institutional ranking, but will rapidly be repurposed in order to shape the behaviour and priorities of academics. For example, the participation of local areas (POLAR) classification allows universities to be rated according to their success against the Widening Participation (WP) agenda. In this way, universities can appear to fail by revealing larger differential outcomes for target groups according to ethnicity and social class than their benchmark permits. The discourse of the TEF legislation, bolstered by studies from HEA/HEPI, assumes the source of inequality of outcome is poor teaching and requires corrective action by universities. Further justification for surveillance and quasi-regulation is borne by appeals to ‘value for money’ and ‘competition’. Universities are positioned as subject to market forces, and students positioned as consumers. Universities are responding by creating ‘managers for the student experience’ whose responsibility it will be to oversee change, without ever addressing the question of what causes differential outcomes, or what actions on behalf of government or institutions might make a difference.

I argue that what seems to be an arbitrary constellation of proxy data points has in fact been a calculated plan to render universities, staff and students as neoliberal subjects. The accident of accessibility, inasmuch as it overlaps with the neoliberal imperative, has determined which data shall function as Key Performance Indicators (KPIs). These KPIs are signalled via metrics-driven student and staff dashboards which offer no retreat from the interpretation and coding imposed by government, and the whole assemblage is cemented by discursive choices which align with neoliberal principles. In this way, the ideological purpose of the legislation and the audit is realized: the imposition of institutional and personal responsibility for structural inequality has been achieved.

The Government White Paper Success as a Knowledge Economy, May 2016, will form the text for corpus analysis of keywords, discourses and metaphors.

The Disrespect of the TEF

 

I have been a few days late to the sector-wide freakshow that is the TEF results.  There has been a news-grabbing but probably temporary perturbation to the traditional hierarchy of mostly English universities, their VCs enlivened by the prospect of being able to raise tuition fees. The resulting categorisations of Gold, Silver and Bronze brought forth just one expression of outrage from Sir Christopher Snowden, the VC of the University of Southampton, and former Universities UK president. Altogether three more (Liverpool, Durham and York) have joined him in launching an appeal. Endorsed by the vice-president of the NUS, Sorana Vieru, Snowden levelled this criticism, “I know I am not alone in having deep concerns about its subjective assessment, its lack of transparency, and with different benchmarks for each institution removing any sense of equity and equality of assessment.”

The statement would have had more force if it had been delivered before the data for the TEF had been sent to Hefce by compliant vice-chancellors. Better still, the sector could have prevented needless and undeserved reputational damage to a majority of universities which received Bronze and Silver ratings if they had stuck together and paid heed to the uncontestable arguments against the TEF. The Council for the Defence of British Universities (CDBU) has published several informative pieces on their website. A key contributor is Professor Dorothy Bishop of the University of Oxford whose argument is summarised in this excellent reply to an article by Edward Peck, VC of Nottingham Trent University.

Her key points are that the justification for the TEF of supposed ‘lamentable teaching’ is unfounded, and in any case, weaknesses in teaching can be diagnosed and addressed by the current QAA inspection and quality framework. Crucially, “The validity of the National Student Survey as a measure of teaching quality has been roundly criticised, and these criticisms appear to have been accepted by the chair of the TEF.” Also, “the statistical properties of NSS data have been described as unsuitable as a measure of teaching quality by the Office for National Statistics, the Royal Statistical Society and, most recently, by Lord Lipsey, chair of the All Party Parliamentary Group on Statistics.”

This should have been a damning enough condemnation to stop the TEF in its tracks before the HEAR Bill could be hustled through parliament before the General Election. It is disturbing that a process which would not get through the first pass of peer review if submitted to an academic journal is now being triumphantly lauded as begetter of a new and ‘disruptive’ hierarchy of universities which better reflects and serves ‘student choice’.

The majority of those who teach in universities are clear that the TEF methodology is fatally flawed, and the results meaningless. There are those in government and the media, though, who still defend its meager claims on credibility. One such is Nick Hillman, Director of HEPI, the Higher Education Policy Institute, a position he assumed after a spell as the Special Advisor to David Willetts when the latter was Minister of State for Higher Education. Nick appears, from my many years of interaction with him on Twitter, to be an amiable and fair minded human being. I appreciate his willingness to engage with critics of the TEF and higher education policy generally. However, I was alarmed at some comments made during this thread. It is worth showing the interaction in full below. The thread starts with the CDBU taking issue with Nick’s view that the TEF offers students important information on which to base their choice of university, a view which echoes the wording of the HE White Paper of May 2016.

What surprised me was Nick’s response to my suggestion that an ill-conceived array of proxy metrics might not serve anybody’s purpose. He seems to believe that it is too much to ask that any policy should work well from Day 1. I suggest that a well-recognised business process, Six Sigma, which focusses on identifying flaws and process improvement, could be adopted here. In an era when public sector organisations are directed to be more business-like in terms of efficiency and effectiveness, I would have thought this would be an article of faith in policy analysis too. Clearly, getting it right in government circles is too much to ask, as Nick reflects on “the messiness of policymaking in a democracy”. Actually, you don’t need anything as systematic as Six Sigma, you just need evidence-based policymaking, and it starts with paying attention to well-informed critics like Dorothy Bishop.

My point in writing this is not to make a personal attack on Nick, who, as I said above, distinguishes himself by his willingness to engage with alternative viewpoints. This is rather a howl of despair at an apparent double standard in expectations between government and those who must deliver their policies. What I am hoping to convey is the frustration of academics who are asked to restrain their own exploratory instincts in favour of a highly-regulated, audited and disciplined approach to working which has predictability and safety as its guiding principles.

What an ironic reversal this is. In previous years we might have expected universities to have license to experiment, to fail, to be messy in their approach. This was how discovery and progress was assumed to move forward. No longer, apparently. While government is afforded the indulgence of failure in policy-making, this is not the case for those who are charged with delivery. For policy makers there appears to be no obligation to embody ‘excellence’, to do what works, to get it right first time, to be evidence- and outcomes-led. These are instead demanded from those who must fulfil them under conditions of an increasingly demanding workload into which more government policy initiatives are emptied on a continuous basis. Unlike government, there is no five-year window for memory slippage when you are subject to performance improvement procedures every three months over some imagined lapse in ‘excellence’. It must be an enjoyable life, driving policy changes when you know that ‘messy’ outcomes will be tolerated, but it is a rather different story in academia 2017.

Let me give just a couple of examples. Research can be a step into unknown territory, but in REF culture, it would be an unwise scholar who set out without a clear sense of – not just results – but the ‘value’ and ‘impact’ of their work. No unpredictable outcomes possible. And similarly, academic staff are expected to fund their research with grant money from research councils with success rates for applications as low as 11%. Similarly, in our teaching, despite recognising that each individual student will take something different from our courses, we must submit them to a system of empty standardized ‘learning outcomes’. And when we have assessed them in a way which belies their imagination and intellectual response, we must endeavor to portray this as ‘personalisation of the learning experience’. There is little tolerance for even necessary ‘messiness’ in academic life. We mourn its passing.

Nick has urged universities to offer more support for students with mental health difficulties in a recent HEPI report. I look forward to another HEPI report which considers the crisis in academic staff mental health, and the role of frameworks such as the TEF and REF in heightening this. They may seem like benign instruments of audit, necessary to justify the considerable public spend in higher education. This would be uncontroversial but for the fact that these have been folded into the disciplinary mechanisms of New Public Management in universities. They too have been personalized and dashboarded into instruments of performance management.  And if HEPI does join the growing band of voices advocating for a more humane university workplace, I hope the report encounters a more gracious reception than I did when I spoke truth to power.

Nick seems to find my retort disrespectful, and if I have been, I offer my apologies. I have always said that the only thing that ever trickles down is contempt, and academics feel it raining down from government, magnified by sections of university management.  As I indicted in my response, democracy has perished in universities alongside ‘messiness’. And when such a double standard is in place, and you are only as good as your last ‘win’, even in the face of structural obstacles, it is nothing less than abusive.

Ten Myths and a Truth from the TEF: Reading the White Paper

Although the Higher Education and Research Bill is still going through parliamentary scrutiny, the Teaching Excellence Framework is about to be implemented and yet we do not know for certain what its effects will be, or even which institutions will enter into it. On the 2^nd of December 2016, the same day as students at Warwick University went into occupation against the TEF , the chair of the TEF, Professor Chris Husbands,  published a blog piece entitled Busting five common myths about the TEF. A welcome addition to the critique, I thought, but I felt as though we were reading different documents.  I have been working on Chapter 2 of the White Paper (TEF) and so I checked some of Jo Johnson’s claims against evidence from some of the other publications I have been reading recently. Concealed within the pages of Jo Johnson’s White Paper, Success as a Knowledge Economy, May 2016,  are quite a few contested propositions and ten more myths which Chris Husbands has overlooked.

We hear much of how political discourse operates in a post-truth culture, but one of the key strategies of persuasion is via presupposition – an statement whose truth is assumed without substantiation. Another trick is to make syntactic linkages between concepts which then acquire the appearance of logical relationship. We find both of these demonstrated in the White Paper.

Below I outline myths (quotations and presuppositions from the White Paper) and responses based on evidence and reason.

Myth 1: There is a problem with ‘lamentable’ teaching quality in universities.

Response: There is no evidence presented to sustain the claim. Use of an inflammatory adjective installs the presupposition.

Myth 2: Students cannot make informed choices….These decisions are significant factors in determining a student’s future life and career success, so it is crucial that they represent sound investments. We need to make sure that students have access to the best possible information to make choices about what they study, and the benefits that they can expect to gain from those choices.

Response: Students have a lot of choice of courses, and they make up their own minds by consulting websites, alternative prospectuses, going to open days. There is even metricised data from Unistats  (comparison site which evaluates NSS scores, employment data and graduate salaries – exactly the innovation Jo Johnson thinks the TEF will deliver) and from league tables.

Nouns like ‘investment’ can also operate as presuppositions as the concept is assumed to be inevitable and universal.  ‘Investment’ is presented in crudely financialised terms as ‘return on investment’ or ROI, which presupposes that students are primarily concerned about future earnings. No evidence is presented to substantiate this, even in the face of students continuing to apply for courses where relatively low salaries are likely upon graduation e.g. nursing, creative arts, education, agriculture. We note that ‘investment’ is a polysemic (multi-meaning) term used to reference the expending of economic capital, and emotional/ intellectual capital by the individual.

Myth 3: Robust, comparable information about the quality of teaching – and the components that contribute to it – is not currently available… That is why this Government will introduce the TEF and for the first time bring sector-wide rigour to the assessment of teaching excellence.

Response: A repetition of the presupposition that students do not already have access to this information. As stated above, it clearly is available. If it is not, why have we been pouring money into QAA, institutional reviews,  Hefce, etc. for all these years, if it has not had the effect of ensuring the quality and reputation of the sector? This architecture of quality assurance, though imperfect, has ensured that the UK is one of the most highly regulated and inspected sectors in the world.

Myth 4: The consumer organisation Which? has found that three in ten students think that the academic experience of higher education is poor value, and the issues raised by students in that research included the amount, and quality, of teaching they received, and the extent to which they are academically challenged.

Response: It is good to see a rare appeal to evidence, but perhaps the wrong conclusions are being drawn by the Which? study. This study by Steven Jones, Steven Courtney and Ruth McGinity proposes another interpretation: “Large fee increases mean that university is bound to be seen as exploitatively expensive by students. This does not mean they are dissatisfied with their courses or teaching quality”. In fact, the NSS scores nationally indicate that students are satisfied with their university experience. Can Jo Johnson make NSS a key metric, and then discount it, all in the same policy document?

Myth 5: Clear priorities of students while at university included: “having more hours of teaching”, “reducing the size of teaching groups” and “better training for lecturers”, but there is little information for prospective students on this in advance.

Response: As this study finds, effective student learning does not always emerge from ‘more contact hours’; in fact independent study is more valuable.   Learning may be the first casualty of a popularity-led evaluation like the NSS/ TEF.

Myth 6: Employers report a growing mismatch between the skills they need and the skills that graduates offer.

Response: A study reported in the Times Higher in 2015 shows that universities are doing a good job in developing the kind of skills which employers find useful and “UK employers are still among the most satisfied with their nation’s higher education system (giving it 7.3 out of 10, compared with a global average of 6.8).”

Myth 7: We need to ensure that our higher education system continues to provide the best possible outcomes. These come from informed choice and competition.

Response: This is a logical non-sequitur, but allows a lazy conflation of several unrelated concepts and assumes causality between them. The White Paper assumes that outcomes = return on investment = graduate salaries, and that these will be consequent upon informed choice and competition. Quality of courses, and choice for students, is more likely to emerge from imaginative cooperation between institutions. This would be an innovation worth pursuing.

This study by David Morris of Wonkhe analyses the government’s Longitudinal Earnings Outcome (LEO) data. There are a number of departures from the outcomes-require-competition myth. Prior attainment, i.e. A Level performance, makes a huge difference to graduate earnings, regardless of subject studied.  This raises a question about ‘learning gain’ – also a concern of the White Paper. I’m sure this will present itself as another cudgel to beat less-favoured universities with. However, Morris’ study also identifies a gender gap and a race gap for earnings, which is far less consonant with a learning gain/ value-added analysis.

Myth 8: By removing student number controls and making it easier for new providers to enter, we will create the conditions that will allow choice and competition to flourish. But what is also needed is the information to allow students to determine where the best teaching can be found.

Response: The answer to quality enhancement, we are expected to believe, is the entry of new providers in order to create ‘competition’. Except the new providers will not be expected to fulfil all the expectations that publically-funded universities are expected to address. As this article makes clear, as new private providers have emerged in strength in South America, especially Argentina and Chile, they have not been engaged in research. This, argues the author – Carolina Guzmán-Valenzuela, restricts the number of qualified PhDs who are able to take the higher education system forward.

Myth 9: The Government believes that excellent teaching can occur in many different forms, in a wide variety of institutions, and it is not the intention of the TEF to constrain or prescribe the form that excellence must take. What we expect though, is that excellent teaching, whatever its form, delivers excellent outcomes.

Response: The TEF will have criteria, and metrics, so how can the White Paper say that the form of excellence will not be prescribed or constrained. In fact, that is exactly what will happen as institutions align their priorities precisely to those criteria – which as the statement makes clear, are in any case based on the proxy ‘outcomes’ of NSS scores, retention and most importantly graduate salaries which are high enough to pay back all the money the government has lost in its ill-advised restructuring of HE finance towards what are, in effect, individual student vouchers.

Myth 10: Perhaps the biggest myth of all – as Jones, Courtney and McGinity point out, is Johnson’s claim that the TEF will strengthen the position of students.  It will not – and indeed, the NUS has voted to disengage from TEF. Evidence shows that co-opting students as consumers is damaging to educational experience.

A truth – a veritable truth: There is of course more to university than financial gain, but the idea that excellent teaching occurs in a vacuum, independent of its impact on students’ future life chances, is not one we can or should accept.

Response:  There is a nice hat tip to other justifications of HE, but immediately we see the counter-narrative remains in place with co-reference of outcomes with financial gain, disguised as ‘life chances’. The presupposition is that the most significant outcome of higher education is employment, but as this study shows, economists have often found that education has benefits for society beyond those of the individual – for example in terms of volunteering, social trust, better citizenship (lower crime).

 

Whatever does ail the higher education sector in the UK, the TEF spreadsheet will not fix it. Much more likely is that the government will recruit ‘consumer choice’ as a disciplinary tool, overlooking the needs of scholarship, local economies or student interests, and possibly serving as licence for university closure. By allowing this false reasoning to go unopposed, we risk losing quality, opportunity and reputation within the sector. Here is a link to the Convention for Higher Education website which has some key resources for opposing the TEF and the Higher Education and Research Bill. Organise, and support students in their refusal to co-operate with the TEF and NSS as long as it threatens to raise their fees, waste millions of pounds of their ‘investment’, threaten the reputation of their courses and distort the priorities of universities away from good teaching and research.

 

TEF Times: 2nd Reading of the HE Bill

In July 2016 we are contemplating a new period of instability for universities in the UK, and with the passing of the 2^nd reading of HE Bill, things could quickly get a lot worse. The EU Referendum result has already created uncertainty regarding the future of much of our research funding. It seems there is much uncertainty at the top of UKHE: Universities UK (‘the definitive voice of UK universities’) has asked for the government to press the pause button on HE reform , Meanwhile, the vice-chancellors of Nottingham Trent and Exeter Universities argue for forging ahead with reform and the implementation of the Teaching Excellence Framework (TEF).

Maddalaine Ansell, CEO of the University Alliance, appears to agree with the latter in her prediction that the HE Bill will take the sector to calmer waters.  Ansell’s premise is that there will be a benefit from having all legislation relating to HE encompassed in one piece of legislation: The Higher Education and Research Act 2017. Except it won’t, of course. A moment’s reflection allows us to list student loans, students with disabilities, and academic freedom – all of which have separate legislation. Additionally, Ansell appears to overlook the added complications of teaching and research which will now be overseen by different government departments since Theresa May’s July ministerial reshuffle. It is a complicated picture, and I cannot see any advantage to deepening it.

Let us remind ourselves just how disruptive these changes proposed by the White Paper entitled Success as a Knowledge Economy (SKE) will be. They include an invitation to new private ‘challenger institutions’ who may be granted degree-awarding powers more quickly than previous regulation allowed. There are changes proposed to governance, academic freedom and protections against arbitrary dismissal which appear to infringe the historic autonomy that universities enjoyed from government. A critique of the proposed changes can be found in an Alternative White Paper (AWP), authored by a group of concerned academics can be found here.

The most unnecessary and wasteful plan in the White Paper is for a Teaching Excellence Framework. This has been proposed to correct supposedly ‘lamentable’ teaching (AWP p28). The paragraphs which outline how this will work display some baffling logical linkages. Here are some of the assertions made in the paragraphs which outline the justification for the TEF:

• higher education leads to better employment outcomes, but these outcomes are not consistent;
• there is considerable variation in employment outcomes and employability amongst subjects and across institutions;
• students often enter HE with little information to guide their choices;
• students often say they would have chosen a different course;
• the importance of students having access to a wide array of work experience opportunities;
• a recent IFS study also found huge variance in graduate earnings depending on choice of subject and institution, as well as background;
• higher average earnings mean that graduates make an important contribution to society through their tax revenues;
• employers and HE providers working together on curriculum design, and graduates having the ‘soft skills’ they need to thrive in the work environment. (SKE p42).

Apparently, the answer to all of these is the TEF which they claim will raise teaching standards. From 2018/19, an award of excellent or outstanding will permit an HEI to increase its fees in line with inflation. Others, even those meeting expectations, will suffer various degrees of attrition and their students condemned to a ‘choice’ of an educational resource eroded by inflation.

The government remains confident that good teaching can be measured on an institutional basis, but the first point to emphasize is that these measures are, as the White Paper admits, proxies, not measures of good teaching which transpires in classrooms and other learning contexts.

“Such things can be measured: students assess their satisfaction with their courses, retention rates are a good proxy for student engagement, contact hours can be measured, employers choose to sponsor some courses, or work with some institutions, because of the industry-relevance of their offerings, and employment rates can be measured. Some of these metrics are of course proxies – but they directly measure some of the most important outcomes that students and taxpayers expect excellent teaching to deliver. And we recognise that metrics alone cannot tell the whole story; they must be benchmarked and contextualised, and considered alongside the additional narrative that can establish a provider’s case for excellence”. (SKE p46)

Secondly, nowhere in the White Paper is there any evidence of so-called lamentable teaching. In fact the published NSS figures show the opposite. Taken nationally, the average figure is extremely high at 86% (England 2015 NSS results) with a rather small range of scores. So why, asks Dorothy Bishop, is there any need for a TEF?

It is hard to avoid the implication that there is likely to be a shift in the direction of prioritising graduate earnings, and indeed, it is one of the proposed measures as the TEF moves towards “a more granular and informative assessment of graduate outcomes” (SKE p48). Possibly the best expose of this misguided proxy measure was the study published in April by the IFS. It demolishes graduate salary as a metric, with its finding that “Graduates from richer family backgrounds earn significantly more after graduation than their poorer counterparts, even after completing the same degrees from the same universities.”  Although this study is acknowledged in SKE, the logic is not absorbed. If we can assume that excellent teaching will not be restricted to more socially advantaged students, what relevance is there to measuring graduate earnings? We can detect an implicit threat in the White Paper that the government may seek to pressure universities to close courses which do not deliver the right ‘outcomes’, i.e. graduates who are able to earn enough to pay back the cost of their student loans. That, then, is the real purpose of this metric. Purely ideological – your graduates don’t pay back – your course is closed.

“In creating the OfS, the regulation of higher education will be restructured, shifting from an outdated, top-down model of a funding agency to a market regulator clearly focused on the student interest. We will give the OfS an explicit duty to promote choice and competition, which will increase quality and efficiency in the sector, and will expect the OfS to work closely with the Student Loans Company and Government to ensure the decisions it takes have regard to affordability and deliver value for money for the taxpayer”. (SKE p63)

In 2017/18 the TEF will be run on a voluntary basis. A ‘provider’ can opt in, presumably if it wishes to establish a good reputation for teaching. A mock league table of benchmark-adjusted metrics published by the Times Higher showed that the Russell Group universities were eclipsed by a Midlands triangle of Loughborough, Aston and De Montfort universities. But this could also be part of the script. The government is creating the conditions whereby the Russell Group flounce out of the TEF and follow the incentives towards privatisation. It is only a matter of time before the elite universities follow their counterparts in Australia and start charging variable fees which will have nothing to do with teaching quality and everything to do with accrued reputation – something which the White Paper claims it wishes to dismantle. Rather than providing concrete information on which students can base their choices, this uninformative snapshot will leave students confused between choosing between the dodgy dossiers of established reputation and the imposter proxies of the TEF.

The TEF will do nothing to increase good teaching, curtail bad teaching or provide students with any more guidance than they already have. And if the REF is anything to go by, it will involve escalating costs and a scale of wastage which makes older, experienced academics weep with regret at what could be achieved if only the money were spent wisely. The cost-benefit analysis is provided by Dorothy Bishop here.

Universities have gone along with the REF because (up to now at least) there were reputational, even if few financial, gains to be won. The TEF allows for little financial gain, and also looks to be repeating some of the reputational mistakes of the early QAA subject reviews which denounced some subjects as failing. The TEF, even when it launches its disciplinary-level ‘granularity’ will not be a ‘game changer’.

Even though universities now have the tools to immediately individualise TEF scores of student satisfaction, nobody is going to be poached by an HEI for their superior teaching scores. Similarly, I would imagine that few academics will be to be tempted to move to a stronger teaching department. And bear in mind, academics have limited agency to affect outcomes such as retention, student satisfaction and employment. Students may be very satisfied with individual teachers, while perceiving elements of the course to be disappointing, funding to be inadequate, accommodation too expensive or the claims of family or paid employment to be stronger.

For universities it is another hurdle to be surmounted. A promised tuition rise in line with inflation will be quickly consumed in the arms race to enhance the institutional image. But the government’s nudge unit will clock up another win as soon as it achieves the desired outcomes; privatisation of an elite tier of universities free to charge whatever they wish, and perhaps, the closure of a few universities which have widened participation, but failed to compensate for the calculated upward distribution of wealth which has been part of the neoliberal project. Whether the HE Bill is creative disruption or reckless joyriding remains to be seen.

While you were away – Summer 2015 HE news catch up Part II

Part II of the digest of HE news, since the UK General Election. This is selective, and there is much I have left out. Below – research, quality assurance, student loans and a glimpse into the post-spending review future.

Research

On the research front, we anticipate the Nurse Review of funding of research. This is being done in the context of an a government spending review, and, BIS has hired McKinsey management consultants. Most pundits suspect this will result in severe cuts to the department’s budget, and it seems likely that all the research funding councils will join together in one body, with the goal of simplifying the system.

This makes everybody nervous – science, because they fear constant chipping away at research funding; humanities and social sciences who fear that they will lose out when funding bids are pitted against the greater claim to economic value attributed to science. Bets are off whether the system of ‘dual funding’ will continue – money from the REF AND money from the research councils. Jo Johnson seems to be suggesting it will in his speech earlier in September 2015. We will know the results on 25^th November, after the government’s spending review has reported.

One piece of unexpected news came with publication in July of the Wilsdon report which came out against a ‘metric tide’ to replace the current REF methodology. Wilsdon’s team found that metrics such as impact factors of journals and citation counts, may easily be gamed, while, by contrast, peer review continues to hold the trust of academics. And David Sweeney, in charge of the REF at HEFCE, seems to agree.

Quality Assurance

Another messy field at the moment. QAA is nearing the end of its contract with HEFCE as regulator of the HE sector. That means it must enter into competition with another two rivals for the task –HEFCE itself who may seek to bring the work in-house, since they no longer have much say in parcelling out funding, except REF (QR) money. However, the latest intel is that both HEFCE and the REF might be axed by BIS

The other candidate for regulator is the HEA whose shaky future might be sustained by a role in quality assurance. It has been rather enfeebled lately, after a series of funding cuts, and is looking for another role besides dispensing credentials to HE practitioners. If it is looking to take a role in the TEF, though, you’d have thought it might have more information about it (and learning gain) on its website.

As the emphasis shifts from process to outcomes (see previous post on TEF and learning gain), it is difficult to predict how this might end up. Jo Johnson has indicated that he would like QA to be ‘light touch’. In saying this, he is probably mindful of the £5M it costs each year, and a diminishing BIS budget. On the other hand, there is an ideological function that underpins QA, the TEF and the REF, and that is to create winners and losers, and perhaps, to provide a means to ‘exit the market’ for unsuccessful ‘providers’, all camouflaged as ‘valuing teaching’ and ‘putting students at the heart of the system’. A re-vamped regulatory framework offers new ways of realising a more Darwinian platform for competition between universities, as does the removal of student number controls (SNC).

And just to stir the hornets’ nest of competition, the government is keen to encourage more private providers to enter the higher education scene. Indeed their access to degree awarding powers may be speeded up, and their Quality Assurance credentials facilitated. “We are a deregulatory government”, Johnson is quoted as saying.

HEFCE, though,  has been consulting on the future approaches to quality assessment in HE.  One thing is clear – neither HEFCE nor Jo Johnson wish to see an increase in grade inflation (see para79) in order for universities to secure student satisfaction or, through crude output measures, to scale university rankings. There may be a new breed of ‘professionalized’ external examiners in order to curb these tendencies.

Student debt

Student debt is still with us, despite calls to follow Germany in abolishing tuition fees. Additionally, the last budget signalled an end to maintenance loans for students from September 2016, and the threshold for repayment has been frozen, rather than hitched to inflation.

If you need to inform yourself about how student loans work – here is the most authoritative voice on the issue, Andrew McGettigan

Headlines from this piece:

• New student loans are not covered by the Consumer Credit Act and interest rates can be set at the discretion of the relevant Secretary of State without the need for new legislation.
• The rate of interest can be at market rates, not at RPI minus 1 percentage point as for the earliest tranche of 1990s student loans (which were protected by the Consumer Credit Act).
• The repayment threshold can also be varied, even after the loan terms have been agreed by the student borrower. The legislation states as follows

“You must agree to repay your loan in line with the regulations that apply at the time the repayments are due and as they are amended. The regulations may be replaced by later regulations.” (p. 8) [http://andrewmcgettigan.org/student-loans-campaign/]

I find it extraordinary, and unethical, to burden 18 year olds with large debts under such uncertain terms.

And the future?

There is a Spending Review in the offing, and George Osborne has already said that austerity is likely to continue for four more years. Sajid Javid, Secretary of State for BIS, the department which deals with universities, is very keen to implement the full 40% cut, which will have a likely impact on HE funding in view of the fact that this spend dominates the department. Not the least of these worries is the RAB charge (the portion of student borrowing that will not be paid back), which has now escalated to a projected figure of 45% of loans.  Fear of that shortfall is evidently what lies behind government attempts to prod universities into channelling their graduates into high-paying jobs. And universities will be happy to comply if they think there’s a league table position at stake.

Additionally, the release of an evaluation of the REF2014 has been put on hold until after the spending review. 77 HEIs offered feedback on their experience of the REF, ‘impact’, cost and interdisciplinarity. You can find your institutional response archived here.

It is a world of financial insecurity, shifting targets and capricious measures of success. I’m having a hard time planning the next 6 months, never mind a curriculum or a research career. Happy new academic year, everybody.

While you were away…Summer 2015 HE news catch up – Part I

Welcome back. That’s assuming you had a holiday in the first place. In case you missed them, here are some of the issues which have emerged since the UK General Election in May (remember that?).

University teaching

You will probably be aware that student number controls have been relaxed from this September. You might imagine this would signal the government’s huge confidence in the university sector. However, the Minister for Higher Education, Jo Johnson, made a speech to the Universities UK Annual Conference on Wednesday 9th September 2015. A useful Bird-and-Fortune-style commentary on the speech can be accessed here. In the speech he announced,

“there is lamentable teaching that must be driven out of our system. It damages the reputation of UK higher education and I am determined to address it”.

This pronouncement reprised some of the themes from his speech at the same venue on July 1st, particularly the idea of a Teaching Excellence Framework.

This was the speech where some additional information was added to the Conservative Manifesto promise to “introduce a framework to recognise universities offering the highest teaching quality”, but we have still to hear what format it might take, and despite much discussion in the media, the focus is still not clearly defined. Nick Hillman has suggested there are three possible candidates: the familiar Quality Assurance process, National Student Survey scores or a measure of ‘learning gain’. Or a mixture of all three, with DLHE (Destinations of Leavers from Higher Education) data thrown in as well.

This is as much as we know so far, from the July 1st speech:

“I expect the TEF to include a clear set of outcome-focused criteria and metrics. This should be underpinned by an external assessment process undertaken by an independent quality body from within the existing landscape”. [http://www.wonkhe.com/blogs/back-to-school-with-jo-johnson/]

The only thing that has been asserted with any clarity is that any increase in the tuition fee will be linked to an institution’s performance in the TEF. In doing so, the government could, at last, claim a success in creating a market in higher education ‘providers’. As we know, the post 2012 reforms had led to a rush for all universities to charge the maximum £9000, or near to it. But now vice-chancellors, particularly in the Russell Group, are lobbying for a fee hike. This could mean some strange incentives arising. If students know that their positive NSS scores will result in a tuition increase for their successors, will they seek to skew their responses, and the university’s league table position, downwards?

Could the TEF resemble a QAA style subject review format? It is hard to imagine that the tried and tested, despised but thoroughly gameable process, will not emerge in some new form. The usual promises have been made, that it would be light touch, and it would need to be, considering the £5 currently spent on QA. However, the QAA was process-focussed, and Jo Johnson has been clear that he wants the emphasis to be on ‘outputs’. In that case, what might it measure?

One candidate is ‘learning gain’ and please see here for a discussion. Simply put, can our graduates demonstrate the nominated transferable skills to a greater extent than before they started higher education? Some commentators talk about ‘distance travelled’.

The OECD just completed a feasibility study into an international comparison of graduates. It seems that European universities are not yet willing to rank their graduates’ learning outcomes against those from other continents. Last week, it was announced that a Europe-only feasibility study will begin: Measuring and Comparing Achievements of Learning Outcomes in Higher Education in Europe project, known as Calohee.

Yet another possibility for the TEF seems to be favoured by Edward Peck, Vice-Chancellor of Nottingham Trent University. He has evidently read this section from the Conservative Manifesto:

“We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality… and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”.

Peck takes that textual linkage of teaching quality and career paths of graduates and makes a learning gain metric out of them. In a piece entitled “Finding new ways to measure graduate success”, he outlines his view thus:

“[T]he forthcoming availability of HMRC tax data to HESA and the Student Loans Company means that we could use a robust measure where we can select the census point at which we present data on average earnings by university and/or by course. This would not be dissimilar to the approach some rankings take to MBA programmes. With secondary education performance data also being brought into the mix, we have the hope of finding a much needed way to measure added value or learning gain”. [http://www.wonkhe.com/blogs/finding-new-ways-to-measure-graduate-success/]

Added value and learning gain are not the same thing, and neither can be measured by graduate salaries. There seemed further valid points to be made against Peck’s suggestion, and so I did, here.

Learning Gain is changing its shape almost daily, and even HEFCE can’t keep up. The Times Higher reports that a number of English (and it is only English) universities are trialling standardized tests from the Wabash National Study.Lots of luck getting your students to turn out for this battery of 13 different tests which have no direct relevance for them.

Meanwhile, BIS is still busy consulting about what the TEF should look like, and at the same time has commissioned a study by RAND Europe into what learning gain is, and how it might be measured. Spoiler – like me, they don’t seem to favour graduate salaries as a valid measure.

Happy new Academic Year. More soon on Research, Quality Assurance, Student debt, and the road ahead in Part II.

The disciplinary dashboard: from reception class to retirement

The photo above made me start contemplating the intrusion of a repressive disciplinary culture into UK universities. Disciplinary action for tailgating? Whatever happened to having a quiet word with somebody? Just a few years ago, campus security was left in the capable hands of a few retirees from the services and the police. They knew academics and students by name, and exerted a calm authority refined through years of dealing with minor infractions. Now, a mere parking violation incurs a meeting with HR.

Many of us will be aware of new university policies on disciplinary procedures. If we have read them, we will be aware that the policies themselves are often not in the least repressive or out of kilter with professional expectations. It is when these policies intersect with over-zealous performance management procedures that things get troublesome – I have previously blogged about so-called under-performing professors 

So when I read the front page of the New York Times this morning (Sunday 16^th August 2015), the portrayal of compulsory overwork and inhumane demands at Amazon in Seattle seemed unsettlingly portentous. When employees ‘hit the wall’ from the unrelenting pace, they are told to ‘climb the wall’. Amazon boasts an approach termed ‘purposeful Darwinism’ which ensures the lowest ranked employees are ‘eliminated’. This is facilitated by an Anytime Feedback Tool – a ‘widget’ which allows co-workers to report each other to management for poor performance or bad attitude. Shockingly, among the victims of this regime, there were employees with long-term serious health problems. According to one interviewee, he had witnessed everybody he worked with breaking down and crying in the office at some point. No wonder.

So this is testosterone-fueled Silicon Valley, not academia. But the future is closer than you think. It is not just a tightening vice around professors and their ‘performance management’, it seems that the panopticon is about to be extended across the whole academic hierarchy with the introduction of ‘faculty dashboards’. These are tools which allow data on each academic to be collated into an individual profile showing publications, citations, research grants and awards won. It can be updated daily by the head of department, dean or vice-chancellor. Norms can be established, and of course, extended year-on-year. They may be changed, according to strategic priorities beyond the control, or indeed the value set, of academics.

This may seem alien and frightening to the current generation of academics. I hope so. What frightens me, is how little resistance this style of management evokes from current undergraduate students. Many universities now have a ‘student dashboard’ apparently aimed at supporting students and increasing retention. It may record VLE logins, door swipes, tutorial attendance, titles of library books borrowed, assignment submissions and grades. When I asked my students if they were comfortable with revealing all this to me, who had just met them, they were nonchalant, and even welcoming of a virtual servo-system which would keep them ‘on track’.

I wonder if this acceptance will be even more enthusiastic among a generation raised with this ‘educational disciplinary system’. Demeritus keeps track of rules, issues penalties, informs parents and, chillingly, discursively inaugurates a new generation of ‘repeat offenders’ – all before they have even learned to ride a bicycle. I rather hope it will inspire sullen resistance if not outright intergenerational retribution.

This disciplinary excess is a sign of a culture which chooses to ‘invest’ in privatized prisons rather than ‘subsidize’ schools and universities. It is certainly familiar in the US to residents of states like Texas, where social justice forums have identified a school-to-prison pipeline.

It is dangerous – immoral – to allow childhood and adolescent transgressions to remain on an electronic rap sheet, to be uncovered when, for what – a job application, adoption process, or even running for President? And when students graduate to college, they face even more repression. Paul Greatrex has written about the routine arming of US university police with military hardware. We have learned this year that such environments may bring about dangerous consequences for students and faculty of color. In the UK too, police have been brought onto campus to quell student protests at the Universities of Birmingham, Warwick and London.

Universities, as I have blogged elsewhere, are unpopular in sections of the media, and with many in the Conservative government. They have come under scrutiny in the US as well, with President Obama and presidential candidate Hillary Clinton both questioning the spiraling cost of higher education. This has occasioned a predictable attack on the easy targets – tenured faculty members. A bill is being considered by the Iowa Senate which purports to relate to the teaching effectiveness and employment of professors. I quote from the first few paragraphs of SF64:

Each institution of higher learning under the board’s control shall develop, and administer at the end of each semester, an evaluation mechanism by which each student enrolled in the institution shall assess the teaching effectiveness of each professor who is providing instruction to the student each semester… Scores are not cumulative. If a professor fails to attain a minimum threshold of performance based on the student evaluations used to assess the professor’s teaching effectiveness, in accordance with the criteria and rating system adopted by the board, the institution shall terminate the professor’s employment regardless of tenure status or contract. (2) The names of the five professors who rank lowest on their institution’s evaluation for the semester, but who scored above the minimum threshold of performance, shall be published on the institution’s internet site and the student body shall be offered an opportunity to vote on the question of whether any of the five professors will be retained as employees of the institution.

Dismissing apparently competent, but unpopular academics starts to look very much like the Amazon ‘purposeful Darwinism’. We can only imagine the consequences for the stability of programs, research and collegiate relations. As we anticipate the arrival of the Teaching Excellence Framework, we must hope that it does not cement a culture of perpetual surveillance and ruinous ‘consumer choice’ by National Student Survey scores. If there is no pause button in academia, if there is no room for slow work, risk, failure and unpopularity, then universities really will have become a disciplinary dystopia.

England Rejects the Learning Tower of Pisa ?

Earlier this month (July 2015), the Times Higher announced the news that England will not be taking part in an OECD project to make Pisa-style international comparisons of graduates’ learning gain. This came as a surprise, given the priority this Conservative government has placed on learning gain and on a Teaching Excellence Framework.

To those of us following the debate, the OECD’s AHELO (Assessment of Higher Education Learning Outcomes) project seemed like the only game in town. Now, apparently we are back to the drawing board on what measures may constitute learning gain and the TEF (see previous post for critique).

The story was followed up in The Guardian on 28^th July.The first thing that struck me was what seemed like a collective sigh of relief by the nation’s vice-chancellors and other HE worthies. This from a group of people who have never met a ‘tool’ they couldn’t beat their staff with. Except this time, the ‘tool’ could not be gamed, or weaponized against academics. It threatened, instead, to appraise the qualities of England’s graduates, and clearly there is an insecurity about how they might hold up in international comparison. After all, the much vaunted REF has managed to avoid any kind of international evaluation, so why not the TEF? Perhaps the vice-chancellors had read a recent report from HEPI indicating that students from other countries work harder than home students. Or perhaps they had also seen pictures of students in Guinea studying outside under the floodlights at the airport, and shuddered at the comparison with pictures of the more bacchanalian evening adventures of UK students.

Maybe they had also sensed disapproval from the Higher Education Minister as he diagnosed grade inflation among the escalating numbers of First and 2.1 degrees awarded. But in any case, this was an unexpected retreat after we had listened for months to all the sermonizing about empowering students to demand better teaching and make universities accountable for the public spend.

Whatever had made the VCs and their friends anxious, it is clear that the TEF may be charting new and uncomfortable ground. UK universities have become really adept at process – what is known lower down the ranks as ‘quality bollocks’. Do your learning outcomes show a hierarchy of cognitive domains? Do they line up with your modes of assessment? Yes, indeed they do, and it has nothing whatsoever to do with teaching quality or learning gain. After several years of this, I am now giving master classes in this to my US colleagues as they enter a new era of fictitious ‘assessment’. In the UK, we have process down to a fine art, but engagement is less of a triumph. This we need to address, since nobody ever signed on to a university course because its quality management processes were optimized in strategic alignment.

If engagement means active, enthusiastic participation in a course leading to measurable learning outcomes (precis of a definition offered in an HEA document)  we have a problem in UK universities. Lecturers complain that attendance is poor on many courses and that students fail to prepare for seminars or read outside of class. There is a culture of permitting students with failed modules to progress to the next level under rules of ‘compensation’. I understand that most students have other responsibilities such as paid work, but still, many graduate knowing they would have got more out of their courses if they had tried harder. This, I think is one reason for the reluctance to enter into international comparisons of learning gain and transferable skills.

But none of this features among the justifications recorded in the Guardian article. On the question of AHELO (Assessment of Higher Education Learning Outcomes), Peter Williams, former CEO of the Quality Assurance Agency is quoted in the hard copy of the article, “I thought the whole thing was a nonsense”. Alison Wolf, of King’s College, London, offers a more cogent evaluation, “It is basically impossible to create complex language-based test items which are comparable in difficulty when translated into a whole lot of different languages”. Comparability is more likely to depend on culture than on language, so given the constituency of the OECD, we can probably reject that claim.

My understanding of the AHELO assessment was that it followed the kind of reasoning, problem-based scenarios presented in the CLA (Collegiate Learning Assessment), so I turned to my colleague Louise Cummings for a view. Louise – a linguist and philosopher – pretty much dominates the field of informal logic and scientific reasoning under conditions of uncertainty (Cummings 2015). Would performance on these tasks be affected by translation from one language to another? No. A firm, no.

So, as I write, the University Alliance (the ‘mission group’ which used to be for ‘business-oriented universities, but now claims “we are Britain’s universities for cities and regions. We believe in making the difference across everything we do”) is holding a day event to discuss the TEF. I’m sure they would prefer to find a way of slipping back into their comfort zone of process-oriented subject review. But if we have to have a TEF, it should be more like the US NSSE (National Survey of Student Engagement) which would put the emphasis where it belongs: on student engagement, and on “how the institution deploys its resources and organizes the curriculum and other learning opportunities to get students to participate”.  I’m not making great claims for this particular instrument, but something similar might give us a picture of how to marry teaching effectiveness and student learning measures.

Cummings, Louise. 2015. Reasoning and Public Health: New Ways of Coping with Uncertainty. London: Springer.