Against ‘Impact’

The other day we interviewed a couple of PhD scholarship candidates. Good applicants, with interesting and unusual research proposals. However, I was saddened when one of them started talking about ‘impact’. So, she’d found out about the institutional regime of evaluation that now governs research and learned the lingo, but is this the game that applicants ought to be playing?

I’ve been thinking about this because I’ve had to write a case-study self-evaluation for the university’s rehearsal for the REF, the administrative-bureaucratic exercise replacing the RAE, that governs the disbursement of £1.6b of research funds in England annually. Under the new rules, 20% of the final score is being awarded for the ‘impact’ of research in the wider world, for which the rubric is ‘an effect/change/benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’. The figure of 20% is of course quite arbitrary, but the real problem is the requirement to quantify what cannot necessarily be quantified, especially in the arts and humanities.

To put it baldly, the requirement for research to demonstrate impact is a crude tool of bureaucratic manipulation that bends and distorts the multiple functions of research towards its own utilitarian purposes. These purposes are not just administrative but also political convenience—the funding body must satisfy the political paymasters. That’s how bureaucracy works, although it mustn’t be too blatant. When the Arts and Humanities Research Council announced almost a year ago that it would spend a considerable part of its £102m funding pot on research into ‘localism’ and the ‘big society’, widespread protests against subjection to a political agenda eventually led to a climbdown [here and here].

According to a leader in the THES in October [here] the current focus on ‘excellence’ stems from Business Secretary Vince Cable’s line that the UK had to stop funding ‘mediocre research’. But the problem here isn’t just what he said and how it gets interpreted. It’s the fact that the minister in charge of higher education is the Business Secretary. How come? It happened on New Labour’s watch, when the first Business Secretary to take charge of universities was Peter Mandelson, the come-back man who was comfortable with the filthy rich as long as they pay their taxes (which many of them don’t). We’re speaking of a managerialist ideology in the interests of commercial and business criteria—criteria essentially alien to the concept and practice of academic freedom. The same ideology that’s intent on ‘reforming’ the whole university system. Anything that can’t be value-costed doesn’t count. Humanist values of inquiry are old hat unless they can be commoditised, or at least conveniently packaged for purposes of cost analysis. If I remain attached to such values, it’s not because I’m an intellectual Luddite, it’s because ideas, arguments, debate, are not commodities, or not reducible to commodities, the material form in which they circulate within the market, but mental or psychological or aesthetic forms whose life is lived outside and beyond the market.

From the perspective of the bureaucracy, this is exactly what’s problematic about the traditions of disinterested learning of the university, whose freedom and autonomy is an essential element of the pursuit of knowledge for its own sake. Hence the new emphasis on ‘robust’ evidence, that is, hard data, that is, figures and numbers. This process ineluctably produces gobbledygook like the warning of a Guardian blogger that ‘There’s still a lot we don’t know about the relative quality profile of the output weightings.’ [here]   There is now even an official sub-branch of the academic research industry devoted to the issue of how to produce robust evidence. It’s called the National Co-ordinating Centre for Public Engagement [here] and it’s funded by JISC, the higher education technology and IT body. Not only that. Universities are spending money on employing specialised administrators to handle the complex process of compiling the required documentation. Academics themselves, meanwhile, as a reader commented on the aforementioned THES leader, are ‘having to waste their time filling in forms and inventing narratives to justify research which has already undergone a peer review process at point of funding’.

However, according to the NCCPE’s deputy director, some academics ‘are struggling to find any evidence, as they just didn’t think to capture it at the time…some will use anecdotal evidence, whereas others will use a more sophisticated way of evidencing their impact, and they will stand shoulders above those who don’t.’ ‘Sophisticated’ here is a euphemism. It means flashy data processing, whose results, when displayed in the form of animated bar graphs and pie graphs and what-have-you, however pretty, is a total abstraction.

From where I sit—the zone of what’s termed research-as-practice or practice-as-research, I don’t know the difference—this seems the height of absurdity, even a kind of mild insanity. You now not only have to do research but you have to do research into your research. Actually, I already do. I take the films I make to screenings and engage in Q&A sessions with the audience, following the practice of the independent film movement back in the 1970s. Then I write it up in a reception diary of some kind which gets published in an academic journal. This is what we academics are supposed to undertake to do to get the funding in the first place, and I always do it. But not in a way likely to please the bureaucrats, because what interests and concerns me is the interpretation, always provisional, of the different, often ambiguous, but felt responses to the film that you encounter in this way.

These are not the kind of questions that can be answered through a list of  quantifiable criteria, because they involve factors like aesthetic and moral judgement, where quantitative evaluation is always reductive and anyway irrelevant. Maybe the findings of the social sciences can sometimes be tracked in the required way, at any rate when they’re taken up by non-academic bodies in the context of policy debates. It is very rare for works of creative practice like video or the performance arts to produce a direct and concrete impact on, say, policy making. But they can and do contribute at the level of communities, both local and virtual, and are often linked to alternative social initiatives and campaigning. The kind of impact they achieve, however, is like all aesthetic experience, of an experiential and existential kind that necessarily falls beneath the bureaucrats’ radar.

In short, the impact of an aesthetic work begins by enhancing and intensifying the beholder’s awareness, and you don’t need a neuroscientist to tell you that (though they might have interesting things to say about it). In an aesthetic sense, the primary moment of impact is the moment of reception, and a video or performance is created for the purpose of arousing the beholder’s immediate response as such, because without it there cannot be any longer-lasting impact. If I’m wrong about this, then I wasted all the time I spent as a student of philosophy and aesthetics.

Then there’s the question of engagement in the digital sphere, which applies to all scholars in the humanities and is much encouraged—only for the evaluators to turn round and say that blogs don’t count (perhaps it’s because they aren’t peer reviewed). Part of the problem is in the mythologies of the digital domain, like the unexamined notion of the viral as a measure of impact when numbers alone tell you nothing and the viral is in fact entirely relative. You can use social media and blogging (like this) to extend the reach of your work, but this doesn’t produce a measurable indicator of impact. As another blogger puts it, ‘social media metrics are often poorly calculated and even more poorly understood’ [here]. Which doesn’t mean your blogging isn’t worth the effort (I hope).

The digital arts face a particular problem, because most of the works produced within the academy inevitably remain marginal, simply because the web is hugely dominated by commercial operators and popular trivia. Nevertheless, it is also very good at letting a thousand flowers bloom in its interstices. Appraising the impact of one these flowers, however, is far from straightforward. For example, such figures as provided by the web platforms that carry the videos are only rough-and-ready guides to dissemination. They may tell you about the distribution of viewers across different countries, but not what kind of viewer they are, even though the video in question is clearly reaching out into the wider world. The reception of streaming video, despite the web’s much touted interactivity, remains anonymous.

This is connected with sociological and conceptual questions about identity which again place the administrative categories in question. The idea that impact outside the academy can be neatly separated out and measured ignores what has been taught within the academy for many years, that everyone’s identity is multiple and shared with others in different groupings. A student shares a video through a social network with family or friends outside it: is she doing so as a student, or as a sister, or as a drinking companion? A university lecturer sends a link to members of his cricket club, some of whom are also academics but in other disciplines: is he doing so as an academic colleague, or a sporting pal, or as a concerned citizen?

The problem is not that targeted funding would improve results but a structural one, in which both intellectual production and social critique are disadvantaged and marginalised. The public sphere is not a homogenous arena but a network of dispersed, unequal and overlapping spaces of communication dominated by the centralised mass media which are now also embedded in the virtual domain of the web. (The web is dominated in turn by gigantic new players like Google, Facebook, Twitter, Amazon and the rest, which serve to extend the reach of pre-existing media while also forcing corporate consolidation, like the recent merger of Penguin and Random House.) But the web, notwithstanding its open and borderless flows, is highly fragmented, and most of the time, rather than overcoming cultural compartmentalisation, it amplifies it.  And while it has indeed opened up new spaces for ideas to circulate more or less freely, which is a good thing, it also reproduces the entrenched anti-intellectual populism of the centralised media.

All of which would mean that ‘impact’ is a specious and entirely ideological concept. In the words of another comment on the THES leader mentioned earlier, a weird dogma to apply to the workings of the intellect, and ‘an excrescence of an ideology that has temporarily established its domination not only in Britain but in the Western world generally’.

By the way, if you’re a non-academic reader, please make yourself known to me, so I can include you in my returns.

This entry was posted in Academic life, Cultural Politics and tagged , , , , . Bookmark the permalink.

5 Responses to Against ‘Impact’

  1. Paula B says:

    Dear Michael,
    Many thanks for this piece. I am a recently graduated ECR and am between a rock and a hard place with this exact issue. My work is becoming recognised as ‘impacting’ on film studies discourse as well as social change in Northern Ireland, but it is the forceful, explicit language focusing on this in job applications that I am finding difficult. I am passionate about the work because I think it is important primarily for society and film and visual culture, but on paper I feel forced into having to appear as an extremely ambitious, high quality, REF-returnable academic machine above and beyond any teaching capability and the social worth of my primary research interests. I’m really just wading through the mire here and am only gaining knowledge on everything, but it seems that not only are there so many of us, but so many of us who are of exceptional quality, that means must be found to devise short lists. I guess the candidate who sparked your thoughts was advised to mention this. It is certainly what I am being advised to state when endlessly applying for jobs at the most difficult stretch in the REF cycle. At our end we don’t have a choice, we have to comply in the hope of getting on the first rung. It is heartening that this is being challenged by someone of your standing.
    Thanks,
    Paula

  2. Dear Michael,

    Well put, indeed. As academics, we have failed to counter the privatisation of higher education, and our union, UCU, has also failed to realise the great danger of this process. First the government (of Labour, at the time!) made education into a commodity by marketing it and forcing HEIs to charge the full rate, and then they continued by putting the market values into REF. There was hardly as substantial opposition to REF, for obvious and complex reasons, but that meant we have played into the hands of the marketisers, and now there seems no way back to liberal education.

    Would Einstein rate at all under ‘Impact’? Would Freud, Webber, Keynes, Russell do any better? and I am not mentioning Marx, who had the kind of impact which is not quite appreciated by REF… I can hardly think of one intellectual whose work is likely to rate high under the real impact criteria. Unless you can show the colour of money in your research, this government does not believe in its importance. In a country with three parties, all of them right wing, all of them inclined against social change, equality and grass-roots democracy, it is hardly surprising that academics have also become part of the bulwark of the middle class against democratic change.
    Many academics have been centrally involved in struggles on the left, but very few have really fought the privatisation of higher education. RAE and REF are examples of us as a community becoming co-opted, acting as part of the controlling interests and hoping that by kow-towing we may come out better than others. RAE and REF have been divisive and corrosive even under Labour, and now under the coalition of the rich and comfortable they are so much more so. But how many of us are ready to realise this, to act on it?

    I think we were silent partners in our own destruction, collaborators in the process of shackling our gifts to the mast of capital. And I am not saying that social impact is not important for me; of course it is, but not the kind of impact REF is searching for (in vain, in most cases…). The advancement of society, the value of knowledge to its healthy operation, the potential of democratic change – this is the kind of impact our systems will squelch and squeeze out.

    Please feel free to disagree with my innate pessimism….

  3. Nili Phillis says:

    Dear Prof. Michael,
    Thank you for your article. I received it from my son who is a PhD student of Film Studies in Glasgow. I am a non academic reader.
    It is good to see an article exposing the absurdity of the present evaluation system in the British universities, although it is depressing to realize what is happening in the academia.
    Several years ago I read an article in the New York Review of Books by a professor from Oxford University entitled “The grim fate of British Universities”. Some of the arguments were similar or the same as yours. Apparently things have not improved, perhaps the opposite is true.
    Thank you,
    Nili Phillis.

  4. chuck kleinhans says:

    I appreciate the larger points here, though being in the US my situation is somewhat different, and I don’t know what most of these acronyms are. But I know that I myself judge some research done by others in my field (media studies) as relatively superficial(the Philosophy of The Matrix, etc.), or redundant (do we really need another study of Alfred Hitchcock’s “genius”?).
    “Impactful” is absurd, especially in the arts and humanities, but don’t we still need to discriminate about the seriousness sor usefulness of research? If so, by what means, and in whose judgement?

  5. Richard Schellhammer says:

    I’m particularly struck by the problem of Education adopting a business model to measure potential outcomes. This is a problem that is, in my opinion, destroying higher education here in the US. Currently, my university is going through the two-year process of renewing our accreditation. As the university hires new personnel to gather mountains of date, as faculty and staff spend countless hours gathering reams of reports (instead of doing the silly work of teaching), and as we have endless meetings about our new Quality Enhancement Project (a model drawn from the business standard of ISO9000), I’m struck by the fusion of business with education. We have to prove that we’re turning out a product. “Students” are reduced to “learners” and “teachers” to “facilitators.” So, on an intellectual assembly line, we facilitators pour smartness into the brains of our learners and make them more smarter — something like that. The more product we create (graduation rates) the better we’ve performed.