Evaluating and evidencing research impact

Building evidence for research impact

 

Photo by Susan Holt Simpson on Unsplash

Having written a post about Research Impact Resource, I thought I would add some more resources related more specifically to evaluating and evidencing research impact.

Designing an impact evaluation

Prof Mark Reed from Fast track impact (an excellent resource for all things impact) recorded a Vlog where he explains how to do an impact evaluation to prove whether or not your research had significant and far-reaching benefits. https://youtu.be/p9J_J-4sqH4 

Mark talks about how to go about designing an evaluation that proves your research has had an impact, that is significant and has reach, and you can prove is linked to your research (attribution). He suggests starting your evaluation design based on research approaches that you are comfortable with, eg. for a statistician this may be a multiple regression analysis, for a humanities scholar this might be about capturing stories. Mark recommends playing to your subject area strengths, the methods that you are already comfortable with.

How do I come up with an effective design? Mark outlines a model for ‘How to evidence impact’ from a forthcoming paper, which includes six evaluation designs:

1. Quantitative, experimental and statistical
2. Theory/logic driven
3. Systems/pathways analysis
4. Case-based and qualitative analysis
5. Creative and participatory
6. Evidence synthesis

How much evidence is enough evidence? Answer is: ‘when it is believable’. Get feedback from your colleagues to find out where there may be be holes in the impact evidence and then add to your design (eg. getting a larger samples or generating an in-depth case study.)

Social impact and social media

A recent paper explores the benefits of researchers sharing not just research findings but also the ‘social impact’ of their research on social media.

‘…the creation of a new method to evaluate the evidence of social impact shared in social media. The evaluation proposed is to measure the social impact coverage ratio (SICOR), focusing on the presence of evidence of social impact shared in social media.’ (Pulido, Redondo-Sama, Sorde´-Martı´and Flecha 2018, page 3)

Social impact in social media: A new method to evaluate the social impact of research (2018) by Pulido CM, Redondo-Sama G, Sorde´-Martı´ T, Flecha R. PLoS ONE 13(8): e0203117. https://doi.org/10.1371/journal.pone.0203117

Impact Evidence

Research Funding and Impact Consultant Stephen Kemp also has a useful website with resources and a blog post with some advice for what testimonials and statements about research impact could include: Guidance on testimonials and statements to corroborate impact from May 2018.

Another useful resource is the journal: Research For All – it is open access journal and has a range of articles including ones about evaluating impact from public engagement and ideas for involving users (often a key pathway to demonstrate impact). The journal is co-sponsored by the UCL Institute of Education and the National Co-ordinating Centre for Public Engagement.

See for example: Copley, J. (2018) ‘Providing evidence of impact from public engagement with research: A case study from the UK’s Research Excellence Framework (REF)’.
Research for All, 2 (2): 230–243.

Advertisements

Research impact resources

The impact unicorn in its natural habitat. (Image by rawpixel.com)

I have been looking at resources in the area of research impact, and as I am in the UK, this means a lot of this is related to the Research Excellence Framework (REF). I wanted to share what I have collated and I will probably be adding to this as I read more.

In the REF draft guidance on submission from July 2018 impact is broadly defined. Here is a snippet:

Impact includes, but is not limited to, an effect on, change or benefit to: the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding of an audience, beneficiary, community, constituency, organisation or individuals in any geographic location whether locally, regionally, nationally or internationally.

I am not sure whether Dr. Bayley coined the phrase ‘impact unicorn’ but it is a great phrase (and I borrowed it for the illustration of this post) and a good reminder that impact comes in all shapes and sizes. See the posts she has written about impact:

• Chasing the ‘impact unicorn’ – myths and methods in demonstrating research benefits by Dr Julie Bayley, 23rd Nov 2017.
• Sausages, unicorns and strip clubs. Or impact: the challenge of connection by Julie Bayley, Oct 4th 2018.

Research Excellence Framework (REF) Impact Toolkit (March 2018) by Helen Tilley, Louise Ball and Caroline Cassidy. Great toolkit for researchers who work in policy contexts.

ESRC Impact toolkit https://esrc.ukri.org/research/impact-toolkit/

Blog post summary sheet by David Evans

Make Your Research Known – 10 Tools to Increase Consumption of Your Research by David Evans, the World Bank blog, 26 Sep 2018. First and foremost the recommendation is to distil your message into two or three bullet points to communicate effectively with people outside your field. Also remember that the main points may need to be different for different audiences.

How was Social Media cited in 2014 REF impact case studies? by Katy Jordan and Mark Carrigan, from the LSE Impact blog, June 6th 2018.

‘Effective engagement is a matter of building relationships over time, as opposed to simply making material public via social media. An excessive fixation on metrics “bakes in” a dissemination model of social media, obscuring the relational value it can create (and the capacity for impact ensuing from this) in pursuit of ever more impressive engagement metrics in spite of the ambiguity about what, if anything, these entail for real world impact.’ (Jordan & Carrigan, 2018)

The “long tail” of research impact is engendered by innovative dissemination tools and meaningful community engagement by Kip Jones and Lee-Ann Fenge, LSE impact blog, 20 Feb 2018. Based on the authors’ article, “Meaningful dissemination produces the “long tail” that engenders community impact”, published in the Qualitative Research Journal (DOI: 10.1108/QRJ-D-17-00037).

‘The fact that the underpinning research began almost a decade and a half ago attests to the principle that research that is meaningful is never really finished, and that dissemination is more than simply a few academic journal articles’ (Kip-Jones & Fenge, 2018)

Developing approaches to research impact assessment and evaluation: lessons from a Canadian health research funder by Julia Langton, from the LSE impact blog, 19th OCt 2018.

McKiernan EC (2017) Imagining the “open” university: Sharing scholarship to improve research and education. PLoS Biol 15(10): e1002614.  https://doi.org/10.1371/journal.pbio.1002614

‘A university’s societal impact depends on the commitment of faculty to transforming their research into reusable information, sharing, and participating in community outreach. As said before, if we want such commitment, universities must develop ways of recognizing and rewarding these activities. Traditional scholarly metrics, like the number of articles published and journal impact factor, give an incomplete picture of true impact. In my opinion, we need a broader perspective’ (McKiernan, 2017)

Kate Williams, Jonathan Grant (2018). A comparative review of how the policy and procedures to assess research impact evolved in Australia and the UK, Research Evaluation, Volume 27, Issue 2, 1 April 2018, Pages 93–105.

ISRIA statement: ten-point guidelines for an effective process of research impact assessment. Health Research Policy and Systems 2018, 16:8. 

Research Impact on Legislatures, Sep 2018.  A briefing note providing an overview of research impact in legislatures. https://www.parliament.uk/research-impact.

See also Research impact at the UK Parliament for resources and training events https://www.parliament.uk/get-involved/research-impact-at-the-uk-parliament/

David Phipps, Joanne Cummings, Debra Pepler, Wendy Craig, and Shelley Cardinal (2016). The Co-produced Pathway to Impact Describes Knowledge Mobilization Processes. Journal of Community Engagement and Scholarship, vol 9, no 1.

For a critical review of the impact agenda see Professor Robert Macdonald’s paper: Robert MacDonald, (2017) ““Impact”, research and slaying Zombies: the pressures and possibilities of the REF”, International Journal of Sociology and Social Policy, Vol. 37 Issue: 11-12, pp.696-710, https://doi.org/10.1108/IJSSP-04-2016-0047

National Co-ordinating Centre for Public Engagement is a great place for resources and they also published a review of how public engagement featured in the case studies submitted for REF 2014.

Impact playbook for Museums, Libraries and Archives by the Europeana Foundation

Emerald Publishing Impact resources https://www.emeraldpublishing.com/resources

UK Research and Innovation on impact https://www.ukri.org/innovation/excellence-with-impact/

Presenting your narrative to demonstrate fit: reflections on applying for jobs and attending interviews

photo by Antony Pilling (all rights reserved)

Between July 2017 and Aug 2018 I have written about fifty applications and attended around ten interviews. I applied mainly to UK universities that were in commuting distance, charity/museum sector, national/international organisational jobs with a focus on research, communication, engagement, academic and research support and project coordination. Here is some of the stuff I have thought about based on a year of job applications and interviews.

Up to date CV

If, like me, you worked for one organisation for a long time [seven years] it can be really difficult to remember all the projects, tasks and changing responsibilities that you have been involved in. But it is important to reflect on these and consider the things you have learnt from some of the situations you have been in, eg. difficult negotiations, giving feedback, team communication skills, working with a variety of organisations/stakeholders/external partners etc.

Applications

All organisations are different and require different type of applications, and this is even more true if you are applying for jobs in different countries. Do the research to find out what the expectations are for a CV and application. If you are applying to a UK University, most of the time I would say that you need to make sure that you provide at least one example for each criteria. I also tended to include a short paragraph at the end about why I am interested in the role.

I found motivated cover letters very difficult to write and I am not sure I ever completely got the hang of them as they are in addition to a CV. But I did feel the best ones I wrote contained details of what I would bring to the role and about my personal motivation. But as I never got shortlisted for interview for any of the roles that required a cover letter, perhaps that is an indication that there is more work to do. It is difficult to say as those organisations requiring cover letters either sent very basic emails to say the job was filled or never got back to me about the outcome.

Writing more or less the same application for similar roles/criteria to different organisations can result in being shortlisted or not. All the things you cannot control, such as amount and quality of other candidates, play a huge role and you can only do your best application/interview for any given opportunity.

I asked my friends to read through numerous applications to give me feedback and this was always really helpful – also sometimes they suggested better examples as they knew work I had done. I can’t thank my friends enough for supporting me with the job search.

Interviews

Research your interview panel if you get the details of who they are. Look up their professional profile and social media presence.

This is really obvious but you need to prepare example answers for interview questions. A good rule is to prepare answers that have examples structured, for example following a STAR structure. Situation, Task, Action and Result. This allows you to develop structured and compelling answers where you outline the situation, detail the task required, the action YOU took and the outcome as result. If like me you find it difficult to constantly foreground yourself and your contribution then try and think about it as if you are talking about someone else BUT you do need to say ‘I’ a lot, as in I did this, I had this responsibility, I contributed this etc.

Prepare answers to the following questions:

• What motivated you to apply for the role and what skills and experience do you think you would bring to the position?
• Can you give us an example of when you have worked effectively as a member of a team?
• Can you give us an example of when you worked independently on a project/task and with a tight deadline?
• Can you give us an example of when you had to deliver some negative/critical feedback and how you approached this situation?
• Can you give us an example of when you have had to use your initiative [to make a decision, to accomplish a task etc]?
• How would you anticipate establishing and maintaining effective communication channels [with colleagues in other departments, external partners etc]?
• How would you handle conflicting deadlines for tasks? Or how would you prioritise a complex workload? It is likely there will be a question about how you organise/approach your work/projects.

Even though I am not sure I think it is quite fair on applicants, as they only really know what is in the job specification, I have realised that it is a very good idea to present a narrative about how you will approach working in the role. So if the role requires communication via social media, develop a communications strategy with details of channels and pros and cons. Presenting a written version at the interview of what you have prepared is a good idea. I know that a lot of interviews give you a task to prepare and this may include aspects of this. This is about presenting yourself and your skills/experience/knowledge in relation to what they want the role to achieve. A lot of this may well be hypothetical as you don’t know how the role will be. It is about creating a narrative with details that demonstrates the fit between you and them.

If you are asked to do a presentation and you are using power point slides, then bring a print out for each of the panel members so they have a copy to scribble on. Also bring any other relevant examples of your work and give the panel one copy.

Do try and think about one or two questions that you want to ask, you may weave them into the main bit of the interview, but it is your opportunity to steer a discussion. If it is a new role you could ask about the context for the role being created. If it is a permanent role you could ask about any professional development opportunities that are available.

Feedback

Always ask for feedback after an interview. Yes, I know it is no fun but most of the time it can be useful, maybe even in ways you hadn’t thought about. One organisation gave me the feedback that my presentation style was too informal and this just confirmed for me that this was not an environment I wanted to work in.

Liminal space in The Breakfast Club

In-between spaces by Kathrine Jensen     CC BY-NC-SA

Anyone who reads my infrequent blog post will have noticed that I go on about liminality and liminal spaces. I am always considering what might be a good example of a liminal space and at some point this month, it struck me that the 1985 movie The Breakfast Club is an excellent example. For me liminal space can include physical space, digital space, any interactive or conversational space or a combination of these. Crucially, it is a space that allows us to step outside ‘normal’ roles/activities/structures and represent an opportunity to explore, reflect, negotiate and the potential for transformation and change.

As the wikipedia entry on Liminality says:

‘During liminal periods of all kinds, social hierarchies may be reversed or temporarily dissolved, continuity of tradition may become uncertain, and future outcomes once taken for granted may be thrown into doubt’

For anyone who hasn’t seen The Breakfast Club the plot is loosely that a group of five students are in detention in the school library [!] on a Saturday and is asked by the teacher to produce an essay on ‘who you think you are’. The students are from different cliques and at the beginning of the movie are presented as five stereotypes:

• Claire – the popular rich girl
• Andrew – the sports jock
• Brian – the geek
• Allison – the outsider
• John – the delinquent

As the day progresses they begin to come together as a group and they emerge as individuals and break out of their stereotypes. This happens through conversation, physical confrontation, sharing the content of their wallets/bags, comparing their packed lunches (or lack of), breaking the rules together when they leave the library and importantly sharing their stories about why they are in detention. Vernon, the teacher, who stands in for clueless adults, also functions as a way to bind the group together in joint resistance.

So here is some of the ways I think The Breakfast Club is an interesting tool to think about what happens in liminal spaces:

• The space itself is not neutral but it does put them all in the same position to some extent – Claire and Andrew are still portrayed as privileged in a number of ways, but they are subject to the same conditions as the others.
• Through talking about their lives, they all end up reflecting on/questioning their place/role in the societal structures they are expected to fit into.
• They use the space to question their actions and motives and to what extent these reflect who they are, who others expect them to be and/or who they want to be.
• The characters experience a sense of freedom in the space [or that is how I interpret the scenes where they play music, dance, etc.] even if they acknowledge this could be temporary.
• It is portrayed as a space for negotiation [of identity and more] and a place of transition where we can see ‘becoming’ happening.
• It is clear that being in a liminal space is both an experience that leaves them vulnerable, scared and exposes them but at the same time it can also empower them to explore their identity and break out of expected behaviours.

At the end of the movie, the characters recognise that the time [and space] they have had together has allowed them to see each other/themselves differently and break away from the rigid hierarchical interactions. But what happens when this time-out comes to an end? They ask themselves: will they still be friends – if they are now. Will they say hello to each other? What ridicule and social pressure from their established cliques (if they are part of one) will they risk? They acknowledge that some of them have more to loose than other, in terms of prestige and position.

They are asking whether the dissolution of the social hierarchies can be sustained outside of the space they find themselves in. I think the movie ends on a hopeful note; that what the students have experienced in the liminal space means that they have changed and will not blindly continue in the same way as before. I would argue this is illustrated by the match making where our intial expectations of who might end up as couples are reversed as the Sports jock ends up with the Outsider and the Delinquent ends up with the Popular girl [significantly Claire gives John one of her diamond earrings at the end of the movie – I always thought this was a very symbolic ‘distribution of wealth’ gesture].

I have more thoughts about the fact that the space is a library, the things that are in the library space and how they students use the space but that is for another post perhaps.

The #altc alphabet

Photo by Pixabay on Pexels.com

Following the excellent example of Sheila MacNeill and her blogpost on 25 things I’m looking forward to at #altc here are some of the conference things that I am excited about. The Association for Learning Technology Conference takes place in Manchester, 11-13 Sep:

• a is for access and the very useful pre-conference online space that ALT has organised
• b is for being able to attend the conference. Thanks to University of Sheffield for funding my attendance. I will be mentioning the journal I work on so this is your advance notice that I’ll suggest you take a look at Online Information Review. The journal publishes research on the social, political and ethical aspects of emergent digital information practices and platforms
• c  is for Dr. Catherine Cronin. The conference will be a great opportunity to catch up on all the exciting things she has done after finishing her PhD. For example the co-creation of Equity Unbound – an emergent, collaborative curriculum which aims to create equity-focused, open, connected, intercultural learning experiences
• d is for Dr. Donna Lanclos. I’m excited to be able to connect with Donna in Manchester and learn more about the Digital Perceptions tool –  a tool aimed at getting people thinking about their digital identity. It was developed by Donna, Lawrie Phipps and Zac Gribble
• e is for the excellent supportive community feel that the conference always has
• f is for Dr. Frances Bell who, along with Catherine Cronin, is presenting a session called ‘A personal, feminist and critical retrospective of Learning (and) Technology, 1994-2018’ and I am really looking forward to their reflections
• g is for getting to see Dr. Chrissi Nerantzi. Chrissi co-developed #LTHEchat and there is a special #altc edition of the chat running on Tuesday 11th September
• h is for Huddersfield and the opportunity to catch up with my ex-colleagues from the University of Huddersfield, Dr. Liz Bennett and Dr. Sue Folley. Check out their session, ‘Getting to grips with Learner Dashboards: a research informed critical approach to understanding their potential’
• i is for all the intriguing tweets that I will see, which will give me ‘alternative session envy’
• j is for joining in as many of the opportunities the programme affords
• k is for the amazing keynotes, eg. Dr. Tressie McMillan Cottom. I follow her on twitter and am excited about seeing her IRL. It will be a real treat to hear her critical perspectives on what is going on in higher education. LowerEd. How for profit colleges deepen inequality in America
• l is for learning analytics (LA) and the really interesting sessions on how students engage and perceive LA in this year’s programme. 
• m is for Manchester, a great city for a conference and with lots of things to see and do (plus eat & drink). Check out the Beyond the Conference – what’s on in Manchester? blog post by Dr. Frances Bell
• n is for networking, which the programme leaves plenty of spaces for
• o is for all the open education experiences being shared at the conference 
• p is for the pack of playing cards that participants will get when registering
• q is for all the questions that will be asked in the sessions
• r is for Research in Learning Technology, the excellent ALT journal
• s is for the stickers with #femedtech that I have heard Dr. Maren Deepwell will be bringing
• t is for technology and student and staff partnership approaches.
• u is for the umbrella I should probably pack for going to Manchester but will forget all about
• v is for the venue, University of Manchester
• w is for the wildcard presentations, I think that is a really great way to accommodate exciting work that may not fit the themes
• x is for the [e]xhibitors and perusing what they have to offer
• y is for Youtube, where you view the keynotes if you are not able to make it to the conference [this one is the same as Sheila’s – it was just too good]
• z is for the [bu]zzwords in ALTc Bingo

Sharing university press practices – our initial findings

Sharing university press practices – our initial findings

https://hudunipress.wordpress.com/2018/09/05/sharing-university-press-practices-our-initial-findings/
— Read on hudunipress.wordpress.com/2018/09/05/sharing-university-press-practices-our-initial-findings/

Engaging and supporting an Open Access scholarly community

It is always nice when something you have worked on for a while comes together. The University of Huddersfield Press Manager, Megan Taylor, asked me to co-author a paper based on her work on developing the processes and impact of the University Press and our collaboration around Fields, a student research journal, published by the Press. Importantly, the University Press is Open Access, and we both believe this has been key to the reach, in terms of downloads and citations, that the student research journal and the other Press publications have had.

Initially, we presented these ideas at the Northern Collaboration Conference in 2017 with our talk on Embracing Open Access publishing for academic staff and student research. To prepare for the presentation we asked our network what open access publishing meant to them and used this for discussion on the day (see image at top of the post).

Then Megan was asked to contribute to a special issue of Publications on ‘Open Access and the Library’ and asked me to co-author the paper building on the work we had done.

Enhancing content to develop engagement

When Megan and I collaborated on Press publications, we were focused, not only on developing excellent content, but very much on involving authors in ways to promote engagement with their final publications. In the paper, we go through a number of the ways that the Press activities engage with authors and editors and the benefits of the different social media platforms in promoting the research of students and staff. The WordPress blog, in particular, ‘proved ideal for creating a space where researchers can tell our readers more about their research in an informal discussion style, whilst always linking back to the original underpinning publication to help drive downloads and, ultimately, citations’ (Taylor and Jensen 2018)

References

Taylor, Megan and Jensen, Kathrine S.H. Engaging and Supporting a University Press Scholarly Community. Publications. 2018, 6, 13. doi:10.3390/publications6020013

Taylor, Megan and Jensen, Kathrine (2017) Embracing open access publishing for academic staff and student research. In: Northern Collaboration 2017 Conference, 08/09/2017, York, England. http://eprints.hud.ac.uk/id/eprint/33203/