Metric Tide Revisited

The independent report The Metric Tide (TMT) was released in 2015 and its recommendations outlined a way of using research metrics responsibly in the design of the REF. It led many of us to hope it would curb some of the more damaging effects of metrics in research assessment.

To a large extent, REF panels are now aware that the focus must be on peer review, supplemented by responsible metrics. Although the Stern Review agreed with the recommendations of TMT, there have been some failures of implementation, acknowledged by the report’s author, James Wilsdon. On reflection, he feels that TMT was overly managerial in its approach, and there needs to be a focus on changing the culture towards healthier research processes.

There was definitely a moment of optimism after the 2014 REF that universities may be induced into treating research, the process of research and researchers with more respect and support. Together with the publication of TMT, the movement for responsible metrics has been buoyed by some declarations of principle such as the Leiden Manifesto for Research Metrics and the San Fransisco Declaration on Research Metrics (DORA) . Despite this, we have seen some egregious and irresponsible abuses of metrics, even among the many UK universities which have become signatories. Some credit the REF with having driven some more undesirable results outcomes including an increased homogenisation in research (see work by Juan Pablo Pardo-Guerra) and the never-ending fixation with rankings.

UKRI has now decided it is time to review how the purposes and design of the REF could be improved. This discussion and consultation is known as the Future Research Assessment Programme (FRAP)  which will “look afresh at the role of metrics in any future research excellence framework and consider whether design changes now under consideration as part of the FRAP suggest similar or different conclusions to those reached in 2015”. The review is being led by Stephen Curry, James Wilsdon and Lizzie Gadd, so in my estimation, the project is in good hands.

There are a number of consultations for FRAP taking place, and at a Zoom round table on July 12th, Silke Machold, Dean of Research of the University of Wolverhampton, wondered how to change behaviours which distort research priorities, but which nevertheless construct the criteria for success. The platforms which provide the research monitoring extract huge profits while funding for research is impoverished. Machold questioned what value is created by metrics and the associated requirements to monitor, report and manage.

We also heard from Rachel Gooberman-Hill, Chair of the UK Committee on Research Integrity, set up this spring with the intent to champion rigour, transparency and care and respect for researchers. We all raise our hats to that, but the next speaker, Patricia Murray of the University of Liverpool, illustrated how far we are from those ideals. Disgracefully, metrics such as grant capture and field-weighted citations have been used to select academics for redundancy at her university . Another low point for integrity was the shameless targeting of scholars in critical management studies at Leicester – a discipline which no longer exists there. And there have been multiple reports of academics being served notice of redundancy after successful REF results in their units of assessment. As Catherine Davies of the University of Leeds advised, breaches of DORA by signatories need to be addressed before cynicism takes root. Unless we have managers who embody research integrity in their own practice, then metrics will continue to be weaponized against researchers and academic freedom further undermined.

I agree that the REF has driven research culture, so if it is to continue (and it will), we must identify how we want that culture to change. So here is a suggestion. The one area which seems most open to modification is research environment. It is the area most focussed on people rather than figures, so any transformation could start with an assessment of the experience of academics themselves – postgraduate, post-doctoral and senior researchers. There are many tools like Vitae-CEDARS but again, these are expensive and complex corporate assessments. What is more appropriate is a simpler ‘satisfaction’ questionnaire including questions on perceptions of academic freedom, respect and support for research. This might deliver the kind of picture required, along with some metrics such as staff continuity and turnover, indicators of environment for mental health and, as well as PhD ‘throughput’, a question about the jobs that doctoral graduates go on to.

Another point. The various elements of the REF do not all need to be assessed concurrently. The environment measure surely is amenable to being uncoupled from the one-time census of research outputs and impact. Universities are apparently committed to rolling REF assessment and frequent mock exercises, and have put in place all the infrastructure to manage them. In that case, asking them to send updates on the research environment shouldn’t be burdensome. Continuing REF funding should be contingent on maintaining a healthy research environment that demonstrates standards of integrity and care for research and researchers. It would stop unscrupulous management teams from taking academics’ work for the REF census and then making them redundant the next day. We might find that their commitment to research as teamwork becomes more than mere window-dressing. It would redress the imbalance of power created when the Stern Review allowed institutions to submit outputs created at the institution after a researcher had left or retired. As well as injecting some much-needed integrity, it would give a more realistic picture of UK higher education as a research environment.

Anyone wanting to add their own ideas for the conduct of the next REF can email the team at responsiblemetrics@gmail.com

Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s