Tag Archives: National Student Survey

TEF Times: 2nd Reading of the HE Bill

In July 2016 we are contemplating a new period of instability for universities in the UK, and with the passing of the 2nd reading of HE Bill, things could quickly get a lot worse. The EU Referendum result has already created uncertainty regarding the future of much of our research funding. It seems there is much uncertainty at the top of UKHE: Universities UK (‘the definitive voice of UK universities’) has asked for the government to press the pause button on HE reform , Meanwhile, the vice-chancellors of Nottingham Trent and Exeter Universities argue for forging ahead with reform and the implementation of the Teaching Excellence Framework (TEF).

Maddalaine Ansell, CEO of the University Alliance, appears to agree with the latter in her prediction that the HE Bill will take the sector to calmer waters.  Ansell’s premise is that there will be a benefit from having all legislation relating to HE encompassed in one piece of legislation: The Higher Education and Research Act 2017. Except it won’t, of course. A moment’s reflection allows us to list student loans, students with disabilities, and academic freedom – all of which have separate legislation. Additionally, Ansell appears to overlook the added complications of teaching and research which will now be overseen by different government departments since Theresa May’s July ministerial reshuffle. It is a complicated picture, and I cannot see any advantage to deepening it.

Let us remind ourselves just how disruptive these changes proposed by the White Paper entitled Success as a Knowledge Economy (SKE) will be. They include an invitation to new private ‘challenger institutions’ who may be granted degree-awarding powers more quickly than previous regulation allowed. There are changes proposed to governance, academic freedom and protections against arbitrary dismissal which appear to infringe the historic autonomy that universities enjoyed from government. A critique of the proposed changes can be found in an Alternative White Paper (AWP), authored by a group of concerned academics can be found here.

The most unnecessary and wasteful plan in the White Paper is for a Teaching Excellence Framework. This has been proposed to correct supposedly ‘lamentable’ teaching (AWP p28). The paragraphs which outline how this will work display some baffling logical linkages. Here are some of the assertions made in the paragraphs which outline the justification for the TEF:

  • higher education leads to better employment outcomes, but these outcomes are not consistent;
  • there is considerable variation in employment outcomes and employability amongst subjects and across institutions;
  • students often enter HE with little information to guide their choices;
  • students often say they would have chosen a different course;
  • the importance of students having access to a wide array of work experience opportunities;
  • a recent IFS study also found huge variance in graduate earnings depending on choice of subject and institution, as well as background;
  • higher average earnings mean that graduates make an important contribution to society through their tax revenues;
  • employers and HE providers working together on curriculum design, and graduates having the ‘soft skills’ they need to thrive in the work environment. (SKE p42).

Apparently, the answer to all of these is the TEF which they claim will raise teaching standards. From 2018/19, an award of excellent or outstanding will permit an HEI to increase its fees in line with inflation. Others, even those meeting expectations, will suffer various degrees of attrition and their students condemned to a ‘choice’ of an educational resource eroded by inflation.

The government remains confident that good teaching can be measured on an institutional basis, but the first point to emphasize is that these measures are, as the White Paper admits, proxies, not measures of good teaching which transpires in classrooms and other learning contexts.

“Such things can be measured: students assess their satisfaction with their courses, retention rates are a good proxy for student engagement, contact hours can be measured, employers choose to sponsor some courses, or work with some institutions, because of the industry-relevance of their offerings, and employment rates can be measured. Some of these metrics are of course proxies – but they directly measure some of the most important outcomes that students and taxpayers expect excellent teaching to deliver. And we recognise that metrics alone cannot tell the whole story; they must be benchmarked and contextualised, and considered alongside the additional narrative that can establish a provider’s case for excellence”. (SKE p46)

Secondly, nowhere in the White Paper is there any evidence of so-called lamentable teaching. In fact the published NSS figures show the opposite. Taken nationally, the average figure is extremely high at 86% (England 2015 NSS results) with a rather small range of scores. So why, asks Dorothy Bishop, is there any need for a TEF?

It is hard to avoid the implication that there is likely to be a shift in the direction of prioritising graduate earnings, and indeed, it is one of the proposed measures as the TEF moves towards “a more granular and informative assessment of graduate outcomes” (SKE p48). Possibly the best expose of this misguided proxy measure was the study published in April by the IFS. It demolishes graduate salary as a metric, with its finding that “Graduates from richer family backgrounds earn significantly more after graduation than their poorer counterparts, even after completing the same degrees from the same universities.”  Although this study is acknowledged in SKE, the logic is not absorbed. If we can assume that excellent teaching will not be restricted to more socially advantaged students, what relevance is there to measuring graduate earnings? We can detect an implicit threat in the White Paper that the government may seek to pressure universities to close courses which do not deliver the right ‘outcomes’, i.e. graduates who are able to earn enough to pay back the cost of their student loans. That, then, is the real purpose of this metric. Purely ideological – your graduates don’t pay back – your course is closed.

“In creating the OfS, the regulation of higher education will be restructured, shifting from an outdated, top-down model of a funding agency to a market regulator clearly focused on the student interest. We will give the OfS an explicit duty to promote choice and competition, which will increase quality and efficiency in the sector, and will expect the OfS to work closely with the Student Loans Company and Government to ensure the decisions it takes have regard to affordability and deliver value for money for the taxpayer”. (SKE p63)

In 2017/18 the TEF will be run on a voluntary basis. A ‘provider’ can opt in, presumably if it wishes to establish a good reputation for teaching. A mock league table of benchmark-adjusted metrics published by the Times Higher showed that the Russell Group universities were eclipsed by a Midlands triangle of Loughborough, Aston and De Montfort universities. But this could also be part of the script. The government is creating the conditions whereby the Russell Group flounce out of the TEF and follow the incentives towards privatisation. It is only a matter of time before the elite universities follow their counterparts in Australia and start charging variable fees which will have nothing to do with teaching quality and everything to do with accrued reputation – something which the White Paper claims it wishes to dismantle. Rather than providing concrete information on which students can base their choices, this uninformative snapshot will leave students confused between choosing between the dodgy dossiers of established reputation and the imposter proxies of the TEF.

The TEF will do nothing to increase good teaching, curtail bad teaching or provide students with any more guidance than they already have. And if the REF is anything to go by, it will involve escalating costs and a scale of wastage which makes older, experienced academics weep with regret at what could be achieved if only the money were spent wisely. The cost-benefit analysis is provided by Dorothy Bishop here.

Universities have gone along with the REF because (up to now at least) there were reputational, even if few financial, gains to be won. The TEF allows for little financial gain, and also looks to be repeating some of the reputational mistakes of the early QAA subject reviews which denounced some subjects as failing. The TEF, even when it launches its disciplinary-level ‘granularity’ will not be a ‘game changer’.

Even though universities now have the tools to immediately individualise TEF scores of student satisfaction, nobody is going to be poached by an HEI for their superior teaching scores. Similarly, I would imagine that few academics will be to be tempted to move to a stronger teaching department. And bear in mind, academics have limited agency to affect outcomes such as retention, student satisfaction and employment. Students may be very satisfied with individual teachers, while perceiving elements of the course to be disappointing, funding to be inadequate, accommodation too expensive or the claims of family or paid employment to be stronger.

For universities it is another hurdle to be surmounted. A promised tuition rise in line with inflation will be quickly consumed in the arms race to enhance the institutional image. But the government’s nudge unit will clock up another win as soon as it achieves the desired outcomes; privatisation of an elite tier of universities free to charge whatever they wish, and perhaps, the closure of a few universities which have widened participation, but failed to compensate for the calculated upward distribution of wealth which has been part of the neoliberal project. Whether the HE Bill is creative disruption or reckless joyriding remains to be seen.

Advertisements

While you were away…Summer 2015 HE news catch up – Part I

Welcome back. That’s assuming you had a holiday in the first place. In case you missed them, here are some of the issues which have emerged since the UK General Election in May (remember that?).

University teaching

You will probably be aware that student number controls have been relaxed from this September. You might imagine this would signal the government’s huge confidence in the university sector. However, the Minister for Higher Education, Jo Johnson, made a speech to the Universities UK Annual Conference on Wednesday 9th September 2015. A useful Bird-and-Fortune-style commentary on the speech can be accessed here. In the speech he announced,

“there is lamentable teaching that must be driven out of our system. It damages the reputation of UK higher education and I am determined to address it”.

This pronouncement reprised some of the themes from his speech at the same venue on July 1st, particularly the idea of a Teaching Excellence Framework.

This was the speech where some additional information was added to the Conservative Manifesto promise to “introduce a framework to recognise universities offering the highest teaching quality”, but we have still to hear what format it might take, and despite much discussion in the media, the focus is still not clearly defined. Nick Hillman has suggested there are three possible candidates: the familiar Quality Assurance process, National Student Survey scores or a measure of ‘learning gain’. Or a mixture of all three, with DLHE (Destinations of Leavers from Higher Education) data thrown in as well.

This is as much as we know so far, from the July 1st speech:

“I expect the TEF to include a clear set of outcome-focused criteria and metrics. This should be underpinned by an external assessment process undertaken by an independent quality body from within the existing landscape”. [http://www.wonkhe.com/blogs/back-to-school-with-jo-johnson/]

The only thing that has been asserted with any clarity is that any increase in the tuition fee will be linked to an institution’s performance in the TEF. In doing so, the government could, at last, claim a success in creating a market in higher education ‘providers’. As we know, the post 2012 reforms had led to a rush for all universities to charge the maximum £9000, or near to it. But now vice-chancellors, particularly in the Russell Group, are lobbying for a fee hike. This could mean some strange incentives arising. If students know that their positive NSS scores will result in a tuition increase for their successors, will they seek to skew their responses, and the university’s league table position, downwards?

Could the TEF resemble a QAA style subject review format? It is hard to imagine that the tried and tested, despised but thoroughly gameable process, will not emerge in some new form. The usual promises have been made, that it would be light touch, and it would need to be, considering the £5 currently spent on QA. However, the QAA was process-focussed, and Jo Johnson has been clear that he wants the emphasis to be on ‘outputs’. In that case, what might it measure?

One candidate is ‘learning gain’ and please see here for a discussion. Simply put, can our graduates demonstrate the nominated transferable skills to a greater extent than before they started higher education? Some commentators talk about ‘distance travelled’.

The OECD just completed a feasibility study into an international comparison of graduates. It seems that European universities are not yet willing to rank their graduates’ learning outcomes against those from other continents. Last week, it was announced that a Europe-only feasibility study will begin: Measuring and Comparing Achievements of Learning Outcomes in Higher Education in Europe project, known as Calohee.

Yet another possibility for the TEF seems to be favoured by Edward Peck, Vice-Chancellor of Nottingham Trent University. He has evidently read this section from the Conservative Manifesto:

“We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality… and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”.

Peck takes that textual linkage of teaching quality and career paths of graduates and makes a learning gain metric out of them. In a piece entitled “Finding new ways to measure graduate success”, he outlines his view thus:

“[T]he forthcoming availability of HMRC tax data to HESA and the Student Loans Company means that we could use a robust measure where we can select the census point at which we present data on average earnings by university and/or by course. This would not be dissimilar to the approach some rankings take to MBA programmes. With secondary education performance data also being brought into the mix, we have the hope of finding a much needed way to measure added value or learning gain”. [http://www.wonkhe.com/blogs/finding-new-ways-to-measure-graduate-success/]

Added value and learning gain are not the same thing, and neither can be measured by graduate salaries. There seemed further valid points to be made against Peck’s suggestion, and so I did, here.

Learning Gain is changing its shape almost daily, and even HEFCE can’t keep up. The Times Higher reports that a number of English (and it is only English) universities are trialling standardized tests from the Wabash National Study.Lots of luck getting your students to turn out for this battery of 13 different tests which have no direct relevance for them.

Meanwhile, BIS is still busy consulting about what the TEF should look like, and at the same time has commissioned a study by RAND Europe into what learning gain is, and how it might be measured. Spoiler – like me, they don’t seem to favour graduate salaries as a valid measure.

Happy new Academic Year. More soon on Research, Quality Assurance, Student debt, and the road ahead in Part II.

The disciplinary dashboard: from reception class to retirement

The photo above made me start contemplating the intrusion of a repressive disciplinary culture into UK universities. Disciplinary action for tailgating? Whatever happened to having a quiet word with somebody? Just a few years ago, campus security was left in the capable hands of a few retirees from the services and the police. They knew academics and students by name, and exerted a calm authority refined through years of dealing with minor infractions. Now, a mere parking violation incurs a meeting with HR.

Many of us will be aware of new university policies on disciplinary procedures. If we have read them, we will be aware that the policies themselves are often not in the least repressive or out of kilter with professional expectations. It is when these policies intersect with over-zealous performance management procedures that things get troublesome – I have previously blogged about so-called under-performing professors 

So when I read the front page of the New York Times this morning (Sunday 16th August 2015), the portrayal of compulsory overwork and inhumane demands at Amazon in Seattle seemed unsettlingly portentous. When employees ‘hit the wall’ from the unrelenting pace, they are told to ‘climb the wall’. Amazon boasts an approach termed ‘purposeful Darwinism’ which ensures the lowest ranked employees are ‘eliminated’. This is facilitated by an Anytime Feedback Tool – a ‘widget’ which allows co-workers to report each other to management for poor performance or bad attitude. Shockingly, among the victims of this regime, there were employees with long-term serious health problems. According to one interviewee, he had witnessed everybody he worked with breaking down and crying in the office at some point. No wonder.

So this is testosterone-fueled Silicon Valley, not academia. But the future is closer than you think. It is not just a tightening vice around professors and their ‘performance management’, it seems that the panopticon is about to be extended across the whole academic hierarchy with the introduction of ‘faculty dashboards’. These are tools which allow data on each academic to be collated into an individual profile showing publications, citations, research grants and awards won. It can be updated daily by the head of department, dean or vice-chancellor. Norms can be established, and of course, extended year-on-year. They may be changed, according to strategic priorities beyond the control, or indeed the value set, of academics.

This may seem alien and frightening to the current generation of academics. I hope so. What frightens me, is how little resistance this style of management evokes from current undergraduate students. Many universities now have a ‘student dashboard’ apparently aimed at supporting students and increasing retention. It may record VLE logins, door swipes, tutorial attendance, titles of library books borrowed, assignment submissions and grades. When I asked my students if they were comfortable with revealing all this to me, who had just met them, they were nonchalant, and even welcoming of a virtual servo-system which would keep them ‘on track’.

I wonder if this acceptance will be even more enthusiastic among a generation raised with this ‘educational disciplinary system’. Demeritus keeps track of rules, issues penalties, informs parents and, chillingly, discursively inaugurates a new generation of ‘repeat offenders’ – all before they have even learned to ride a bicycle. I rather hope it will inspire sullen resistance if not outright intergenerational retribution.

This disciplinary excess is a sign of a culture which chooses to ‘invest’ in privatized prisons rather than ‘subsidize’ schools and universities. It is certainly familiar in the US to residents of states like Texas, where social justice forums have identified a school-to-prison pipeline.

It is dangerous – immoral – to allow childhood and adolescent transgressions to remain on an electronic rap sheet, to be uncovered when, for what – a job application, adoption process, or even running for President? And when students graduate to college, they face even more repression. Paul Greatrex has written about the routine arming of US university police with military hardware. We have learned this year that such environments may bring about dangerous consequences for students and faculty of color. In the UK too, police have been brought onto campus to quell student protests at the Universities of Birmingham, Warwick and London.

Universities, as I have blogged elsewhere, are unpopular in sections of the media, and with many in the Conservative government. They have come under scrutiny in the US as well, with President Obama and presidential candidate Hillary Clinton both questioning the spiraling cost of higher education. This has occasioned a predictable attack on the easy targets – tenured faculty members. A bill is being considered by the Iowa Senate which purports to relate to the teaching effectiveness and employment of professors. I quote from the first few paragraphs of SF64:

Each institution of higher learning under the board’s control shall develop, and administer at the end of each semester, an evaluation mechanism by which each student enrolled in the institution shall assess the teaching effectiveness of each professor who is providing instruction to the student each semester… Scores are not cumulative. If a professor fails to attain a minimum threshold of performance based on the student evaluations used to assess the professor’s teaching effectiveness, in accordance with the criteria and rating system adopted by the board, the institution shall terminate the professor’s employment regardless of tenure status or contract. (2) The names of the five professors who rank lowest on their institution’s evaluation for the semester, but who scored above the minimum threshold of performance, shall be published on the institution’s internet site and the student body shall be offered an opportunity to vote on the question of whether any of the five professors will be retained as employees of the institution.

Dismissing apparently competent, but unpopular academics starts to look very much like the Amazon ‘purposeful Darwinism’. We can only imagine the consequences for the stability of programs, research and collegiate relations. As we anticipate the arrival of the Teaching Excellence Framework, we must hope that it does not cement a culture of perpetual surveillance and ruinous ‘consumer choice’ by National Student Survey scores. If there is no pause button in academia, if there is no room for slow work, risk, failure and unpopularity, then universities really will have become a disciplinary dystopia.

National Student Survey: the view from different disciplines

Yesterday (12th August 2015) saw the publication of the UK National Student Survey, spawning much discussion on Twitter with the hashtag #nss.  Here are just a few views collated from academics on Twitter. There’s a bit of editorial licence, of course, in the general spirit of gaming the exercise. On occasions this extends to actual substitution of quotations.

Here are some typical views from the different academic disciplines.

Statistics:  Celebrating a statistically insignificant change in historically unreliable data rendered invalid by spurious comparisons of massively different respondent pools.

Discourse analysis: Signs are small, measurable things; interpretations are illimitable (George Eliot).

Literature: If you can meet with triumph and disaster, and treat those two imposters just the same (Kipling).

Politics: The only statistics you can trust are those you falsified yourself (Churchill).

History: The problem with surveys as indicators of satisfaction is that they only capture the present.

Education: Universities with high NSS scores are failing to teach critical thinking.

Philosophy: Experience cannot be converted to a median score.

Associate Dean: Only a moron would treat this datagarbage seriously. Wait….5th nationally? Wow, seriously? Yay !