The government’s teaching excellence framework, or Tef as it’s usually known, is supposed to eventually measure the quality of teaching in each subject at every university. The idea is that the data will help future students make improved choices about where to study and that it will incentivize better, more transparent behavior from universities. These are all laudable aims, but what if the tool isn’t right for the job?

There is a real risk that the Department for Education’s statistically inadequate approach will lead to distorted results, misleading rankings and a system which lacks validity and is unnecessarily vulnerable to being gamed.

The original Tef evaluated the whole institution, and our response at the time was critical of its methodology. But the new subject-level Tef introduces further issues [pdf]. In trying to allocate each subject a bronze, silver or gold has introduced more complexity to the operation, which makes gaming more likely. It will also rely on smaller, more uncertain sample sizes.

The Tef doesn’t work because it assesses what we can, rather than what we should, measure. It incorporates employment metrics even though graduate jobs may be a result of family background and income rather than university teaching. It also uses the numbers of students per class to capture teaching intensity, which is too simplistic. And it’s too reliant on the National Student Survey, which doesn’t measure teaching, but rather how students generally feel about their course, which can be affected by a wide range of factors. For instance, London universities have lower scores which are often attributed to the cost of living in the capital. The NSS is also an unstable data source. t is also a potentially unstable data source – In 2017 around 25 student unions boycotted the NSS, leading to the data is invalid in a number of universities.

So the Tef is measuring things which have a limited relationship with “teaching excellence”. Further, there will be a lot of statistical uncertainty surrounding any given TEF score, making comparisons (a primary purpose of the whole exercise) close to useless. A new book by Professor Jerry Muller argues that we are seeing the rise of metric fixation and that this inevitably leads to gaming of the system. The architects of the Tef are not particularly mindful of how the system leaves itself open to being played. We have seen this in many other areas of public policy. Think of NHS waiting targets where the National Audit Office found many hospitals under-recording wait times. Or school league tables, where teachers have focused on getting D-grade students up to C grades to the exclusion of the needs of other students – as this is what makes the difference for the league table score.

Nobody knows how much the entire Tef exercise will cost. Jo Johnson, the minister who introduced the Tef, said he wanted to avoid it being a “big, bossy and bureaucratic” process. But partly in response to the criticism that the metrics underlying the Tef are poor, new qualitative panels have been introduced, pushing the costs up. Meanwhile, the main reason many universities adopted Tef was that the government-linked it to increases in student fees. This idea has now been ditched, so there is less of an incentive for universities to take part.

In the long run, the public learns which data they can trust – and which they can’t. Over time, people will realize that Tef doesn’t really tell us very much and will lose trust in the numbers. In the meantime, the whole exercise will waste enormous amounts of time, energy and funding.

There are lessons from history. The teaching quality assessment (TQA) was a project run over a decade to judge the quality of teaching across universities. It was abandoned in 2001, and was widely thought not to have been worth the cost in time or resources. “The dons have outsmarted the government by turning the exercise into a game and playing it brilliantly,” said Prof Geoffrey Alderman, a former TQA assessor, when it ended.

That’s not to say that students don’t need better data on universities so that they can make better choices. We could tell them about the quality of the teaching environment – including class sizes, IT provision, libraries – but we should not pretend this is measuring teaching excellence. To do that properly would be a seriously expensive exercise, using unannounced observation of classes, similar to Ofsted in schools, and looking at how each institution embeds mechanisms for accrediting, assessing and rewarding teaching.

The new universities minister, Sam Gyimah, has nothing personally invested in this system. It’s rare for ministers to have a policy option that would save money, be popular, and be the right thing to do. Not proceeding further with the Tef is one such rare opportunity. The minister should take it.

• Hetan Shah is the executive director of the Royal Statistical Society