Tag Archives: quality assurance

Firsts Among Equals? Why have the number of first class degrees increased so dramatically?

“Figures on degree scores from the Higher Education Statistics Agency, analysed by the Press Association, show that 40 higher education institutions saw the proportion of firsts rise by more than 10 percentage points between 2010-11 and 2015-16” (Simon Baker, Times Higher, July 20th 2017). https://www.timeshighereducation.com/news/sharp-increase-first-class-degrees-triggers-standards-debate

Once again the annual HESA figures on degree classifications have set in motion an outpouring of consternation from broadsheets and tabloid newspapers alike. Photos two weeks ago of conventionally attractive, white, and mostly female students tossing their mortarboards are now displaced by sneers and insinuations that their achievements have been compromised by a wanton lowering of standards. Simon Baker’s piece shows some startling figures: The University of Surrey’s proportion of first class graduates has doubled between 2011 and 2017 from 19.3% to 41.2%, while firsts at the University of East Anglia have tripled from 12.5% to 34%. The article offers some explanations for the large increase: better teaching, higher entry standards to universities, students working harder, the effect of students paying higher fees; league tables which reward high percentages of ‘good degrees’. This year there is an additional incentive to send students out with a first or upper second class degree – many employers demand them, and the TEF rewards institutions with solid track records of graduate employment. But for the newspapers, I suppose it is what they call ‘low hanging fruit’ – something they understand so little about, it is easy to distort the facts and take a whack at universities, ever a target for an eye-bulging populist. You almost expect to see Lord Adonis weighing in.

I think I can offer some insight into why the achievement of students has risen in the past 6 or 7 years. For over twenty years I was actively involved in quality exercises at subject, department and institutional level. I also gained considerable experience as an external examiner. Progress in academic standards and quality has been in process throughout my career, and it seemed to me we had achieved peak criterion-referenced scrupulosity. No longer is the award of a first class degree treated as some kind of mysterious alchemy, recognised only by possessors of an equal or superior first class mind. The culture of metrics and key performance indicators has made itself known to students in the form of learning outcomes and assessment criteria. This means that students are not in the dark about ‘what the lecturer wants’, or ‘what do I need to do to get a first’. The approach to advanced learning that elicits such questions is, to my mind, inherently flawed, but nevertheless, that’s where we are in UK universities. The system has constructed a student who is a consumer with anxieties which must be allayed by the provision of roadmaps to success.

We have also been led in this direction by very sound pedagogy. We now know that the best way to learn is by having a go, and then getting advice from someone whose knowledge and experience can help you improve. To that end, students are given many forms of feedback during their course, among them formative and summative feedback. An assessment may be broken down into, for example, a proposal which is marked and returned with comments (formative), and then an essay or project which a student can write up in the light of this feedback (summative). Lecturers are also encouraged to make available ‘exemplars’ which show how a previous student has attempted the exercise (anonymised, and with permission, of course). These will be accompanied with a commentary which clarifies exactly how the student has met the marking criteria, and why the grade was achieved.

The marking criteria themselves are extremely explicit, and the marker will indicate, for each of the criteria, exactly which level has been achieved. The comments will give guidance on how the student could improve to the next level. This is how quality feedback should work – the student knows exactly where the areas of strength and weakness are, and can work to address them. No surprise then, if more students learn to follow these recommendations in order to achieve the higher grades.

And for the marker too, there are surprises. Sometimes the mark you originally had in your head after reading an essay will change once you start to systematically align your marks with the criteria.

Additionally, for many years external examiners have been encouraging internal markers to extend the range of marks given. It is easy to see how many more students ended up with a 2.2 or a 2.1 when the vast majority of their marks over three years were between 50-69%. If they were awarded a first class mark, this would probably be between 70-74% so it was unlikely to tilt the average over the first class threshold. More recently, the top end of the scale has been opened up because – well, not all first class papers are equally good. So now a student may receive 96% for an exceptional piece which is regarded by markers as almost of publishable standard. This would be very rare. But there can be marks in the 80% range for excellent work, as well as a solid 70+% for the very good pieces. A few of these stellar grades, and you will get more clear, numerical firsts emerging.

Even in pre-HESA, NSS and TEF days, it was never the case that all first were awarded on the basis of reaching the numerical threshold of 70%. This was largely to take into account the rather parsimonious award of marks over 70% for essays and exams, and the artificial compression of those first class marks into the 70-75% range. Very few candidates would end their studies with an aggregated mark of over 70%, so exam boards would consider a candidate for ‘promotion’ to a higher grade if their score fell within 1-2% of the first class borderline. Also, scores were weighted to reflect ‘exit velocity’, so the student who improved over the course of their degree and achieved mostly first class marks in their final year, would graduate with the higher classification. There were other locally-agreed regulations, but as far as I am aware, most universities acted to make sure students were awarded a degree classification which reflected their ability and scholarly improvement. This worked well.

The problem, if it is a problem, is that we now have two forces of uplift operating at the same time: the broadening of the first class marking range, and the regulations for ‘promotion’ to the higher degree classification. If both are applied, it is not surprising that 20-25% of candidates qualify for a first. This would explain the very large increase from 2010/11 to 2016/17.

I am not making any recommendations here, merely trying to add some clarity and reason to what has become a rather volatile issue. I have avoided the term ’grade inflation’ because this oversimplifies the confluence of processes and rationales which have led to the current situation. Universities must prioritise good practice and fairness in teaching, learning and assessment. But they must also guard against the more perverse incentives presented by consumerism, student satisfaction, league tables and the TEF.

 

 

Ten Myths and a Truth from the TEF: Reading the White Paper

Although the Higher Education and Research Bill is still going through parliamentary scrutiny, the Teaching Excellence Framework is about to be implemented and yet we do not know for certain what its effects will be, or even which institutions will enter into it. On the 2nd of December 2016, the same day as students at Warwick University went into occupation against the TEF , the chair of the TEF, Professor Chris Husbands,  published a blog piece entitled Busting five common myths about the TEF. A welcome addition to the critique, I thought, but I felt as though we were reading different documents.  I have been working on Chapter 2 of the White Paper (TEF) and so I checked some of Jo Johnson’s claims against evidence from some of the other publications I have been reading recently. Concealed within the pages of Jo Johnson’s White Paper, Success as a Knowledge Economy, May 2016,  are quite a few contested propositions and ten more myths which Chris Husbands has overlooked.

We hear much of how political discourse operates in a post-truth culture, but one of the key strategies of persuasion is via presupposition – an statement whose truth is assumed without substantiation. Another trick is to make syntactic linkages between concepts which then acquire the appearance of logical relationship. We find both of these demonstrated in the White Paper.

Below I outline myths (quotations and presuppositions from the White Paper) and responses based on evidence and reason.

Myth 1: There is a problem with ‘lamentable’ teaching quality in universities.

Response: There is no evidence presented to sustain the claim. Use of an inflammatory adjective installs the presupposition.

Myth 2: Students cannot make informed choices….These decisions are significant factors in determining a student’s future life and career success, so it is crucial that they represent sound investments. We need to make sure that students have access to the best possible information to make choices about what they study, and the benefits that they can expect to gain from those choices.

Response: Students have a lot of choice of courses, and they make up their own minds by consulting websites, alternative prospectuses, going to open days. There is even metricised data from Unistats  (comparison site which evaluates NSS scores, employment data and graduate salaries – exactly the innovation Jo Johnson thinks the TEF will deliver) and from league tables.

Nouns like ‘investment’ can also operate as presuppositions as the concept is assumed to be inevitable and universal.  ‘Investment’ is presented in crudely financialised terms as ‘return on investment’ or ROI, which presupposes that students are primarily concerned about future earnings. No evidence is presented to substantiate this, even in the face of students continuing to apply for courses where relatively low salaries are likely upon graduation e.g. nursing, creative arts, education, agriculture. We note that ‘investment’ is a polysemic (multi-meaning) term used to reference the expending of economic capital, and emotional/ intellectual capital by the individual.

Myth 3: Robust, comparable information about the quality of teaching – and the components that contribute to it – is not currently available… That is why this Government will introduce the TEF and for the first time bring sector-wide rigour to the assessment of teaching excellence.

Response: A repetition of the presupposition that students do not already have access to this information. As stated above, it clearly is available. If it is not, why have we been pouring money into QAA, institutional reviews,  Hefce, etc. for all these years, if it has not had the effect of ensuring the quality and reputation of the sector? This architecture of quality assurance, though imperfect, has ensured that the UK is one of the most highly regulated and inspected sectors in the world.

Myth 4: The consumer organisation Which? has found that three in ten students think that the academic experience of higher education is poor value, and the issues raised by students in that research included the amount, and quality, of teaching they received, and the extent to which they are academically challenged.

Response: It is good to see a rare appeal to evidence, but perhaps the wrong conclusions are being drawn by the Which? study. This study by Steven Jones, Steven Courtney and Ruth McGinity proposes another interpretation: “Large fee increases mean that university is bound to be seen as exploitatively expensive by students. This does not mean they are dissatisfied with their courses or teaching quality”. In fact, the NSS scores nationally indicate that students are satisfied with their university experience. Can Jo Johnson make NSS a key metric, and then discount it, all in the same policy document?

Myth 5: Clear priorities of students while at university included: “having more hours of teaching”, “reducing the size of teaching groups” and “better training for lecturers”, but there is little information for prospective students on this in advance.

Response: As this study finds, effective student learning does not always emerge from ‘more contact hours’; in fact independent study is more valuable.   Learning may be the first casualty of a popularity-led evaluation like the NSS/ TEF.

Myth 6: Employers report a growing mismatch between the skills they need and the skills that graduates offer.

Response: A study reported in the Times Higher in 2015 shows that universities are doing a good job in developing the kind of skills which employers find useful and “UK employers are still among the most satisfied with their nation’s higher education system (giving it 7.3 out of 10, compared with a global average of 6.8).”

Myth 7: We need to ensure that our higher education system continues to provide the best possible outcomes. These come from informed choice and competition.

Response: This is a logical non-sequitur, but allows a lazy conflation of several unrelated concepts and assumes causality between them. The White Paper assumes that outcomes = return on investment = graduate salaries, and that these will be consequent upon informed choice and competition. Quality of courses, and choice for students, is more likely to emerge from imaginative cooperation between institutions. This would be an innovation worth pursuing.

This study by David Morris of Wonkhe analyses the government’s Longitudinal Earnings Outcome (LEO) data. There are a number of departures from the outcomes-require-competition myth. Prior attainment, i.e. A Level performance, makes a huge difference to graduate earnings, regardless of subject studied.  This raises a question about ‘learning gain’ – also a concern of the White Paper. I’m sure this will present itself as another cudgel to beat less-favoured universities with. However, Morris’ study also identifies a gender gap and a race gap for earnings, which is far less consonant with a learning gain/ value-added analysis.

Myth 8: By removing student number controls and making it easier for new providers to enter, we will create the conditions that will allow choice and competition to flourish. But what is also needed is the information to allow students to determine where the best teaching can be found.

Response: The answer to quality enhancement, we are expected to believe, is the entry of new providers in order to create ‘competition’. Except the new providers will not be expected to fulfil all the expectations that publically-funded universities are expected to address. As this article makes clear, as new private providers have emerged in strength in South America, especially Argentina and Chile, they have not been engaged in research. This, argues the author – Carolina Guzmán-Valenzuela, restricts the number of qualified PhDs who are able to take the higher education system forward.

Myth 9: The Government believes that excellent teaching can occur in many different forms, in a wide variety of institutions, and it is not the intention of the TEF to constrain or prescribe the form that excellence must take. What we expect though, is that excellent teaching, whatever its form, delivers excellent outcomes.

Response: The TEF will have criteria, and metrics, so how can the White Paper say that the form of excellence will not be prescribed or constrained. In fact, that is exactly what will happen as institutions align their priorities precisely to those criteria – which as the statement makes clear, are in any case based on the proxy ‘outcomes’ of NSS scores, retention and most importantly graduate salaries which are high enough to pay back all the money the government has lost in its ill-advised restructuring of HE finance towards what are, in effect, individual student vouchers.

Myth 10: Perhaps the biggest myth of all – as Jones, Courtney and McGinity point out, is Johnson’s claim that the TEF will strengthen the position of students.  It will not – and indeed, the NUS has voted to disengage from TEF. Evidence shows that co-opting students as consumers is damaging to educational experience.

A truth – a veritable truth: There is of course more to university than financial gain, but the idea that excellent teaching occurs in a vacuum, independent of its impact on students’ future life chances, is not one we can or should accept.

Response:  There is a nice hat tip to other justifications of HE, but immediately we see the counter-narrative remains in place with co-reference of outcomes with financial gain, disguised as ‘life chances’. The presupposition is that the most significant outcome of higher education is employment, but as this study shows, economists have often found that education has benefits for society beyond those of the individual – for example in terms of volunteering, social trust, better citizenship (lower crime).

 

Whatever does ail the higher education sector in the UK, the TEF spreadsheet will not fix it. Much more likely is that the government will recruit ‘consumer choice’ as a disciplinary tool, overlooking the needs of scholarship, local economies or student interests, and possibly serving as licence for university closure. By allowing this false reasoning to go unopposed, we risk losing quality, opportunity and reputation within the sector. Here is a link to the Convention for Higher Education website which has some key resources for opposing the TEF and the Higher Education and Research Bill. Organise, and support students in their refusal to co-operate with the TEF and NSS as long as it threatens to raise their fees, waste millions of pounds of their ‘investment’, threaten the reputation of their courses and distort the priorities of universities away from good teaching and research.

 

While you were away – Summer 2015 HE news catch up Part II

Part II of the digest of HE news, since the UK General Election. This is selective, and there is much I have left out. Below – research, quality assurance, student loans and a glimpse into the post-spending review future.

Research

On the research front, we anticipate the Nurse Review of funding of research. This is being done in the context of an a government spending review, and, BIS has hired McKinsey management consultants. Most pundits suspect this will result in severe cuts to the department’s budget, and it seems likely that all the research funding councils will join together in one body, with the goal of simplifying the system.

This makes everybody nervous – science, because they fear constant chipping away at research funding; humanities and social sciences who fear that they will lose out when funding bids are pitted against the greater claim to economic value attributed to science. Bets are off whether the system of ‘dual funding’ will continue – money from the REF AND money from the research councils. Jo Johnson seems to be suggesting it will in his speech earlier in September 2015. We will know the results on 25th November, after the government’s spending review has reported.

One piece of unexpected news came with publication in July of the Wilsdon report which came out against a ‘metric tide’ to replace the current REF methodology. Wilsdon’s team found that metrics such as impact factors of journals and citation counts, may easily be gamed, while, by contrast, peer review continues to hold the trust of academics. And David Sweeney, in charge of the REF at HEFCE, seems to agree.

Quality Assurance

Another messy field at the moment. QAA is nearing the end of its contract with HEFCE as regulator of the HE sector. That means it must enter into competition with another two rivals for the task –HEFCE itself who may seek to bring the work in-house, since they no longer have much say in parcelling out funding, except REF (QR) money. However, the latest intel is that both HEFCE and the REF might be axed by BIS

The other candidate for regulator is the HEA whose shaky future might be sustained by a role in quality assurance. It has been rather enfeebled lately, after a series of funding cuts, and is looking for another role besides dispensing credentials to HE practitioners. If it is looking to take a role in the TEF, though, you’d have thought it might have more information about it (and learning gain) on its website.

As the emphasis shifts from process to outcomes (see previous post on TEF and learning gain), it is difficult to predict how this might end up. Jo Johnson has indicated that he would like QA to be ‘light touch’. In saying this, he is probably mindful of the £5M it costs each year, and a diminishing BIS budget. On the other hand, there is an ideological function that underpins QA, the TEF and the REF, and that is to create winners and losers, and perhaps, to provide a means to ‘exit the market’ for unsuccessful ‘providers’, all camouflaged as ‘valuing teaching’ and ‘putting students at the heart of the system’. A re-vamped regulatory framework offers new ways of realising a more Darwinian platform for competition between universities, as does the removal of student number controls (SNC).

And just to stir the hornets’ nest of competition, the government is keen to encourage more private providers to enter the higher education scene. Indeed their access to degree awarding powers may be speeded up, and their Quality Assurance credentials facilitated. “We are a deregulatory government”, Johnson is quoted as saying.

HEFCE, though,  has been consulting on the future approaches to quality assessment in HE.  One thing is clear – neither HEFCE nor Jo Johnson wish to see an increase in grade inflation (see para79) in order for universities to secure student satisfaction or, through crude output measures, to scale university rankings. There may be a new breed of ‘professionalized’ external examiners in order to curb these tendencies.

Student debt

Student debt is still with us, despite calls to follow Germany in abolishing tuition fees. Additionally, the last budget signalled an end to maintenance loans for students from September 2016, and the threshold for repayment has been frozen, rather than hitched to inflation.

If you need to inform yourself about how student loans work – here is the most authoritative voice on the issue, Andrew McGettigan

Headlines from this piece:

  • New student loans are not covered by the Consumer Credit Act and interest rates can be set at the discretion of the relevant Secretary of State without the need for new legislation.
  • The rate of interest can be at market rates, not at RPI minus 1 percentage point as for the earliest tranche of 1990s student loans (which were protected by the Consumer Credit Act).
  • The repayment threshold can also be varied, even after the loan terms have been agreed by the student borrower. The legislation states as follows

“You must agree to repay your loan in line with the regulations that apply at the time the repayments are due and as they are amended. The regulations may be replaced by later regulations.” (p. 8) [http://andrewmcgettigan.org/student-loans-campaign/]

I find it extraordinary, and unethical, to burden 18 year olds with large debts under such uncertain terms.

And the future?

There is a Spending Review in the offing, and George Osborne has already said that austerity is likely to continue for four more years. Sajid Javid, Secretary of State for BIS, the department which deals with universities, is very keen to implement the full 40% cut, which will have a likely impact on HE funding in view of the fact that this spend dominates the department. Not the least of these worries is the RAB charge (the portion of student borrowing that will not be paid back), which has now escalated to a projected figure of 45% of loans.  Fear of that shortfall is evidently what lies behind government attempts to prod universities into channelling their graduates into high-paying jobs. And universities will be happy to comply if they think there’s a league table position at stake.

Additionally, the release of an evaluation of the REF2014 has been put on hold until after the spending review. 77 HEIs offered feedback on their experience of the REF, ‘impact’, cost and interdisciplinarity. You can find your institutional response archived here.

It is a world of financial insecurity, shifting targets and capricious measures of success. I’m having a hard time planning the next 6 months, never mind a curriculum or a research career. Happy new academic year, everybody.