The impact of academic research, particularly on policy and the private sector, is an increasingly important component of research assessment exercises and funding distribution. However, Duncan Green argues that the way many researchers think about their impact continues to be pretty rudimentary. A lack of understanding of who key decision-makers are, a less-than-agile response to real-world events, and difficulties in attributing credit are all hampering progress in this area. Looking at how impact is measured by aid agencies, there is much academics could learn from their monitoring, evaluation and learning teams.

As someone who works for both Oxfam and the LSE, I often get roped in to discuss how research can have more impact on “practitioners” and policy. This is a big deal in academia; the UK government runs a periodic Research Excellence Framework (REF) exercise, which allocates funds for university research on the basis both of their academic quality and their impact. Impact accounted for 20% last time around, in 2014, and may have an even greater weighting in the next exercise, in 2021. That means hundreds of millions of quid are at stake, so universities are trying to get better at achieving and measuring impact (or at least looking like they are!) as they start to prepare for the next round of submissions.

And they have a way to go, based on my recent experience of listening to pitches from a range of researchers. Some interesting patterns emerged, both in terms of weaknesses and ways to address them.

First up, many academics fall into the classic mistake of equating frenetic activity with impact. Long lists of meetings attended, papers published, speeches given, etc., do not impact make. I think they should start from a different place – how would you set about convincing an intelligent, fair-minded sceptic of the impact of your research?

That might be a challenge because the way a lot of researchers think about their impact seems pretty rudimentary. Finish the research, publish the papers, run a few seminars, produce a policy brief and (at the less stuffy end) bash out some social media, and voila! Unfortunately, “blog it and they will come” is often not a very convincing influencing strategy.

A popular alternative in REF submissions is the adviser/consultancy model – “the UN/Government asked me to present/be on a panel/draft their guidelines – now that’s impact”. A bit more convincing, but both approaches would benefit from a more explicit theory of change, running through some of the following issues:

Stakeholder and power mapping

• For the change you are advocating, who are the likely champions, drivers and, undecideds who you can seek to win over?
• Who has the power over the decisions you are trying to influence? They are probably going to be your main targets.
• Who/what in turn influences those targets? Is it evidence (if so, what kind?) or something else entirely, like the identity of the messenger?
• How do you get targets to be aware of and interested in your research in the first place (since both decision-makers and practitioners live in the land of TL;DR)? Have you tried to involve them in it, by asking them to be on a reference group, comment on drafts, or be interviewed for it, for example? Much better than just adding your paper to their reading pile.

Dynamics

Is the change you seek likely to be smooth or spiky? Some changes are largely crisis-driven, in which case your influencing strategy will have to try to get ready in advance, identify and make the most of crises or other “windows of opportunity”. As Pasteur said, “fortune favours the prepared mind”. This is antithetical to the normal rhythm of research – a steady, high-pressure grind of papers and presentations, largely oblivious to events out there in the world.

Insider vs Outsider strategy

Insiders prize being “in the room”, and see themselves as deeply engaged in policy formulation. Outsiders are more interested in constructing alliances, working with (and providing ammunition for) activists and civil society organisations, and talking to the media. But both need to think through their influencing strategies. On the insider route, are the people in this particular room really the people making the decisions? Which invitations do you say no to (if only to avoid death by consultation)? For outsiders, what are the alliances you need to make in order to reach and influence the decision-makers on this particular issue? In both cases, what kind of research (both in content and presentation) is most likely to get people’s attention?

Attribution

Both approaches struggle with attribution. According to one insider “the ministry is a black box – you’re invited to speak. They say a polite thank you. That’s it.” How do you know if you changed anything? The challenges with outsider strategies are different – in an alliance of a dozen CSOs, think tanks and universities, even if some change has been effected, how do you apportion credit?

The efforts at attribution are currently pretty crude: count the citations and if you think your research has helped someone, you beg them for an email saying just how brilliant you are and how much impact you have had. Better than nothing, but pretty dubious as a sole source of evidence (right up there with job references in terms of rigour….).

There’s a lot that academics could learn from the MEL (monitoring, evaluation and learning) teams in the aid agencies here – they have some incredibly smart people working on this. I’ve seen Oxfam’s Effective Reviews really invest in trying to measure impact by constructing a plausible control group for comparison, through propensity score matching, or comparing and ranking different causal factors through process tracing. The academics I’ve talked to seemed entirely unaware of these kinds of methods. Given the multi-million pound benefits of getting it right, there may be a case for universities to outsource (and pay for) this either to “impact partners” such as NGOs with more experience in the area, or to third-party specialist organisations that could accompany researchers in design for impact, and then measure the results more rigorously.

Then there’s communications. The default tone is neutral and cautious – goals were to “shift” this or “support” that without ever specifying why or towards what. That feels more like avoiding mistakes than communicating a message. One alternative would be to come up with a clear mission statement – “we use evidence to change public opinion so that X happens”. If that’s too close to a credibility-damaging move into overt campaigning, academics should at least be able to clearly identify their USP (unique selling point) – e.g. “I’m the only person in the UK who knows what is going on in these particular far-off places”, and clearly state what their research is about in such a way that normal people and REF panels can understand.

That sounds like an extended moan, but perhaps most striking in the conversations I’ve had is how much impact appears to be taking place anyway – I came away with the impression that a bit more analysis and explicit discussion of assumptions and influencing strategies could be an incredibly productive exercise for universities seeking to turn research into real impact.

This blog post was originally published on the LSE Impact blog, and is an edited version of that which originally appeared on the author’s From Poverty to Power blog. A number of counterpoint comments have been posted underneath the original version, and offer an interesting alternative perspective. The exchanges that followed have been written about in Duncan’s more recent blog post.

Featured image credit: Heartland Archery | Winnipeg by Justin Ladia (licensed under a CC BY 2.0 license).

Duncan Green is Senior Strategic Adviser for Oxfam GB and Professor in Practice in the Department of International Development at LSE. He runs the From Poverty to Power blog and is author of the book How Change Happens. He can be found on twitter @fp2p.