Tag Archives: quality assurance

Ten Myths and a Truth from the TEF: Reading the White Paper

Although the Higher Education and Research Bill is still going through parliamentary scrutiny, the Teaching Excellence Framework is about to be implemented and yet we do not know for certain what its effects will be, or even which institutions will enter into it. On the 2nd of December 2016, the same day as students at Warwick University went into occupation against the TEF , the chair of the TEF, Professor Chris Husbands,  published a blog piece entitled Busting five common myths about the TEF. A welcome addition to the critique, I thought, but I felt as though we were reading different documents.  I have been working on Chapter 2 of the White Paper (TEF) and so I checked some of Jo Johnson’s claims against evidence from some of the other publications I have been reading recently. Concealed within the pages of Jo Johnson’s White Paper, Success as a Knowledge Economy, May 2016,  are quite a few contested propositions and ten more myths which Chris Husbands has overlooked.

We hear much of how political discourse operates in a post-truth culture, but one of the key strategies of persuasion is via presupposition – an statement whose truth is assumed without substantiation. Another trick is to make syntactic linkages between concepts which then acquire the appearance of logical relationship. We find both of these demonstrated in the White Paper.

Below I outline myths (quotations and presuppositions from the White Paper) and responses based on evidence and reason.

Myth 1: There is a problem with ‘lamentable’ teaching quality in universities.

Response: There is no evidence presented to sustain the claim. Use of an inflammatory adjective installs the presupposition.

Myth 2: Students cannot make informed choices….These decisions are significant factors in determining a student’s future life and career success, so it is crucial that they represent sound investments. We need to make sure that students have access to the best possible information to make choices about what they study, and the benefits that they can expect to gain from those choices.

Response: Students have a lot of choice of courses, and they make up their own minds by consulting websites, alternative prospectuses, going to open days. There is even metricised data from Unistats  (comparison site which evaluates NSS scores, employment data and graduate salaries – exactly the innovation Jo Johnson thinks the TEF will deliver) and from league tables.

Nouns like ‘investment’ can also operate as presuppositions as the concept is assumed to be inevitable and universal.  ‘Investment’ is presented in crudely financialised terms as ‘return on investment’ or ROI, which presupposes that students are primarily concerned about future earnings. No evidence is presented to substantiate this, even in the face of students continuing to apply for courses where relatively low salaries are likely upon graduation e.g. nursing, creative arts, education, agriculture. We note that ‘investment’ is a polysemic (multi-meaning) term used to reference the expending of economic capital, and emotional/ intellectual capital by the individual.

Myth 3: Robust, comparable information about the quality of teaching – and the components that contribute to it – is not currently available… That is why this Government will introduce the TEF and for the first time bring sector-wide rigour to the assessment of teaching excellence.

Response: A repetition of the presupposition that students do not already have access to this information. As stated above, it clearly is available. If it is not, why have we been pouring money into QAA, institutional reviews,  Hefce, etc. for all these years, if it has not had the effect of ensuring the quality and reputation of the sector? This architecture of quality assurance, though imperfect, has ensured that the UK is one of the most highly regulated and inspected sectors in the world.

Myth 4: The consumer organisation Which? has found that three in ten students think that the academic experience of higher education is poor value, and the issues raised by students in that research included the amount, and quality, of teaching they received, and the extent to which they are academically challenged.

Response: It is good to see a rare appeal to evidence, but perhaps the wrong conclusions are being drawn by the Which? study. This study by Steven Jones, Steven Courtney and Ruth McGinity proposes another interpretation: “Large fee increases mean that university is bound to be seen as exploitatively expensive by students. This does not mean they are dissatisfied with their courses or teaching quality”. In fact, the NSS scores nationally indicate that students are satisfied with their university experience. Can Jo Johnson make NSS a key metric, and then discount it, all in the same policy document?

Myth 5: Clear priorities of students while at university included: “having more hours of teaching”, “reducing the size of teaching groups” and “better training for lecturers”, but there is little information for prospective students on this in advance.

Response: As this study finds, effective student learning does not always emerge from ‘more contact hours’; in fact independent study is more valuable.   Learning may be the first casualty of a popularity-led evaluation like the NSS/ TEF.

Myth 6: Employers report a growing mismatch between the skills they need and the skills that graduates offer.

Response: A study reported in the Times Higher in 2015 shows that universities are doing a good job in developing the kind of skills which employers find useful and “UK employers are still among the most satisfied with their nation’s higher education system (giving it 7.3 out of 10, compared with a global average of 6.8).”

Myth 7: We need to ensure that our higher education system continues to provide the best possible outcomes. These come from informed choice and competition.

Response: This is a logical non-sequitur, but allows a lazy conflation of several unrelated concepts and assumes causality between them. The White Paper assumes that outcomes = return on investment = graduate salaries, and that these will be consequent upon informed choice and competition. Quality of courses, and choice for students, is more likely to emerge from imaginative cooperation between institutions. This would be an innovation worth pursuing.

This study by David Morris of Wonkhe analyses the government’s Longitudinal Earnings Outcome (LEO) data. There are a number of departures from the outcomes-require-competition myth. Prior attainment, i.e. A Level performance, makes a huge difference to graduate earnings, regardless of subject studied.  This raises a question about ‘learning gain’ – also a concern of the White Paper. I’m sure this will present itself as another cudgel to beat less-favoured universities with. However, Morris’ study also identifies a gender gap and a race gap for earnings, which is far less consonant with a learning gain/ value-added analysis.

Myth 8: By removing student number controls and making it easier for new providers to enter, we will create the conditions that will allow choice and competition to flourish. But what is also needed is the information to allow students to determine where the best teaching can be found.

Response: The answer to quality enhancement, we are expected to believe, is the entry of new providers in order to create ‘competition’. Except the new providers will not be expected to fulfil all the expectations that publically-funded universities are expected to address. As this article makes clear, as new private providers have emerged in strength in South America, especially Argentina and Chile, they have not been engaged in research. This, argues the author – Carolina Guzmán-Valenzuela, restricts the number of qualified PhDs who are able to take the higher education system forward.

Myth 9: The Government believes that excellent teaching can occur in many different forms, in a wide variety of institutions, and it is not the intention of the TEF to constrain or prescribe the form that excellence must take. What we expect though, is that excellent teaching, whatever its form, delivers excellent outcomes.

Response: The TEF will have criteria, and metrics, so how can the White Paper say that the form of excellence will not be prescribed or constrained. In fact, that is exactly what will happen as institutions align their priorities precisely to those criteria – which as the statement makes clear, are in any case based on the proxy ‘outcomes’ of NSS scores, retention and most importantly graduate salaries which are high enough to pay back all the money the government has lost in its ill-advised restructuring of HE finance towards what are, in effect, individual student vouchers.

Myth 10: Perhaps the biggest myth of all – as Jones, Courtney and McGinity point out, is Johnson’s claim that the TEF will strengthen the position of students.  It will not – and indeed, the NUS has voted to disengage from TEF. Evidence shows that co-opting students as consumers is damaging to educational experience.

A truth – a veritable truth: There is of course more to university than financial gain, but the idea that excellent teaching occurs in a vacuum, independent of its impact on students’ future life chances, is not one we can or should accept.

Response:  There is a nice hat tip to other justifications of HE, but immediately we see the counter-narrative remains in place with co-reference of outcomes with financial gain, disguised as ‘life chances’. The presupposition is that the most significant outcome of higher education is employment, but as this study shows, economists have often found that education has benefits for society beyond those of the individual – for example in terms of volunteering, social trust, better citizenship (lower crime).

 

Whatever does ail the higher education sector in the UK, the TEF spreadsheet will not fix it. Much more likely is that the government will recruit ‘consumer choice’ as a disciplinary tool, overlooking the needs of scholarship, local economies or student interests, and possibly serving as licence for university closure. By allowing this false reasoning to go unopposed, we risk losing quality, opportunity and reputation within the sector. Here is a link to the Convention for Higher Education website which has some key resources for opposing the TEF and the Higher Education and Research Bill. Organise, and support students in their refusal to co-operate with the TEF and NSS as long as it threatens to raise their fees, waste millions of pounds of their ‘investment’, threaten the reputation of their courses and distort the priorities of universities away from good teaching and research.

 

While you were away – Summer 2015 HE news catch up Part II

Part II of the digest of HE news, since the UK General Election. This is selective, and there is much I have left out. Below – research, quality assurance, student loans and a glimpse into the post-spending review future.

Research

On the research front, we anticipate the Nurse Review of funding of research. This is being done in the context of an a government spending review, and, BIS has hired McKinsey management consultants. Most pundits suspect this will result in severe cuts to the department’s budget, and it seems likely that all the research funding councils will join together in one body, with the goal of simplifying the system.

This makes everybody nervous – science, because they fear constant chipping away at research funding; humanities and social sciences who fear that they will lose out when funding bids are pitted against the greater claim to economic value attributed to science. Bets are off whether the system of ‘dual funding’ will continue – money from the REF AND money from the research councils. Jo Johnson seems to be suggesting it will in his speech earlier in September 2015. We will know the results on 25th November, after the government’s spending review has reported.

One piece of unexpected news came with publication in July of the Wilsdon report which came out against a ‘metric tide’ to replace the current REF methodology. Wilsdon’s team found that metrics such as impact factors of journals and citation counts, may easily be gamed, while, by contrast, peer review continues to hold the trust of academics. And David Sweeney, in charge of the REF at HEFCE, seems to agree.

Quality Assurance

Another messy field at the moment. QAA is nearing the end of its contract with HEFCE as regulator of the HE sector. That means it must enter into competition with another two rivals for the task –HEFCE itself who may seek to bring the work in-house, since they no longer have much say in parcelling out funding, except REF (QR) money. However, the latest intel is that both HEFCE and the REF might be axed by BIS

The other candidate for regulator is the HEA whose shaky future might be sustained by a role in quality assurance. It has been rather enfeebled lately, after a series of funding cuts, and is looking for another role besides dispensing credentials to HE practitioners. If it is looking to take a role in the TEF, though, you’d have thought it might have more information about it (and learning gain) on its website.

As the emphasis shifts from process to outcomes (see previous post on TEF and learning gain), it is difficult to predict how this might end up. Jo Johnson has indicated that he would like QA to be ‘light touch’. In saying this, he is probably mindful of the £5M it costs each year, and a diminishing BIS budget. On the other hand, there is an ideological function that underpins QA, the TEF and the REF, and that is to create winners and losers, and perhaps, to provide a means to ‘exit the market’ for unsuccessful ‘providers’, all camouflaged as ‘valuing teaching’ and ‘putting students at the heart of the system’. A re-vamped regulatory framework offers new ways of realising a more Darwinian platform for competition between universities, as does the removal of student number controls (SNC).

And just to stir the hornets’ nest of competition, the government is keen to encourage more private providers to enter the higher education scene. Indeed their access to degree awarding powers may be speeded up, and their Quality Assurance credentials facilitated. “We are a deregulatory government”, Johnson is quoted as saying.

HEFCE, though,  has been consulting on the future approaches to quality assessment in HE.  One thing is clear – neither HEFCE nor Jo Johnson wish to see an increase in grade inflation (see para79) in order for universities to secure student satisfaction or, through crude output measures, to scale university rankings. There may be a new breed of ‘professionalized’ external examiners in order to curb these tendencies.

Student debt

Student debt is still with us, despite calls to follow Germany in abolishing tuition fees. Additionally, the last budget signalled an end to maintenance loans for students from September 2016, and the threshold for repayment has been frozen, rather than hitched to inflation.

If you need to inform yourself about how student loans work – here is the most authoritative voice on the issue, Andrew McGettigan

Headlines from this piece:

  • New student loans are not covered by the Consumer Credit Act and interest rates can be set at the discretion of the relevant Secretary of State without the need for new legislation.
  • The rate of interest can be at market rates, not at RPI minus 1 percentage point as for the earliest tranche of 1990s student loans (which were protected by the Consumer Credit Act).
  • The repayment threshold can also be varied, even after the loan terms have been agreed by the student borrower. The legislation states as follows

“You must agree to repay your loan in line with the regulations that apply at the time the repayments are due and as they are amended. The regulations may be replaced by later regulations.” (p. 8) [http://andrewmcgettigan.org/student-loans-campaign/]

I find it extraordinary, and unethical, to burden 18 year olds with large debts under such uncertain terms.

And the future?

There is a Spending Review in the offing, and George Osborne has already said that austerity is likely to continue for four more years. Sajid Javid, Secretary of State for BIS, the department which deals with universities, is very keen to implement the full 40% cut, which will have a likely impact on HE funding in view of the fact that this spend dominates the department. Not the least of these worries is the RAB charge (the portion of student borrowing that will not be paid back), which has now escalated to a projected figure of 45% of loans.  Fear of that shortfall is evidently what lies behind government attempts to prod universities into channelling their graduates into high-paying jobs. And universities will be happy to comply if they think there’s a league table position at stake.

Additionally, the release of an evaluation of the REF2014 has been put on hold until after the spending review. 77 HEIs offered feedback on their experience of the REF, ‘impact’, cost and interdisciplinarity. You can find your institutional response archived here.

It is a world of financial insecurity, shifting targets and capricious measures of success. I’m having a hard time planning the next 6 months, never mind a curriculum or a research career. Happy new academic year, everybody.