How to evaluate excellence in arts-based research

This article first appeared in Funding Insight on 19 May 2016 and is reproduced with kind permission of Research Professional. For more articles like this, visit www.researchprofessional.com.

judgementResearchers, research commissioners, and research funders all struggle with identifying good quality arts-based research. ‘I know it when I see it’ just doesn’t pass muster. Fortunately, Sarah J Tracy of Arizona State University has developed a helpful set of criteria that are now being used extensively to assess the quality of qualitative research, including arts-based and qualitative mixed-methods research.

Tracy’s conceptualisation includes eight criteria: worthy topic, rich rigour, sincerity, credibility, resonance, significant contribution, ethics, and meaningful coherence. Let’s look at each of those in a bit more detail.

A worthy topic is likely to be significant, meaningful, interesting, revealing, relevant, and timely. Such a topic may arise from contemporary social or personal phenomena, or from disciplinary priorities.

Rich rigour involves care and attention, particularly to sampling, data collection, and data analysis. It is the antithesis of the ‘quick and dirty’ research project, requiring diligence on the part of the researcher and leaving no room for short-cuts.

Sincerity involves honesty and transparency. Reflexivity is the key route to honesty, requiring researchers to interrogate and display their own impact on the research they conduct. Transparency focuses on the research process, and entails researchers disclosing their methods and decisions, the challenges they faced, any unexpected events that affected the research, and so on. It also involves crediting all those who have helped the researcher, such as funders, participants, or colleagues.

Credibility is a more complex criterion which, when achieved, produces research that can be perceived as trustworthy and on which people are willing to base decisions. Tracy suggests that there are four dimensions to achieving credibility: thick description, triangulation/crystallization, multiple voices, and participant input beyond data provision. Thick description means lots of detail and illustration to elucidate meanings which are clearly located in terms of theoretical, cultural, geographic, temporal, and other such location markers. Triangulation and crystallisation are both terms that refer to the use of multiplicity within research, such as through using multiple researchers, theories, methods, and/or data sources. The point of multiplicity is to consider the research question in a variety of ways, to enable the exploration of different facets of that question and thereby create deeper understanding. The use of multiple voices, particularly in research reporting, enables researchers more accurately to reflect the complexity of the research situation. Participant input beyond data provision provides opportunities for verification and elaboration of findings, and helps to ensure that research outputs are understandable and implementable.

Although all eight criteria are potentially relevant to arts-based research, resonance is perhaps the most directly relevant. It refers to the ability of research to have an emotional impact on its audiences or readers. Resonance has three aspects: aesthetic merit, generalisability, and transferability. Aesthetic merit means that style counts alongside, and works with, content, such that research is presented in a beautiful, evocative, artistic and accessible way. Generalisability refers to the potential for research to be valuable in a range of contexts, settings, or circumstances. Transferability is when an individual reader or audience member can take ideas from the research and apply them to their own situation.

Research can contribute to knowledge, policy, and/or practice, and will make a significant contribution if it extends knowledge or improves policy or practice. Research may also make a significant contribution to the development of methodology; there is a lot of scope for this with arts-based methods.

Several of the other criteria touch on ethical aspects of research. For example, many researchers would argue that reflexivity is an ethical necessity. However, ethics in research is so important that it also requires a criterion of its own. Tracy’s conceptualisation of ethics for research evaluation involves procedural, situational, relational, and exiting ethics. Procedural ethics refers to the system of research governance – or, for those whose research is not subject to formal ethical approval, the considerations therein such as participant welfare and data storage. Situational ethics requires consideration of the specific context for the research and how that might or should affect ethical decisions. Relational ethics involve treating others well during the research process: offering respect, extending compassion, keeping promises, and so on. And exiting ethics cover the ways in which researchers present and share findings, as well as aftercare for participants and others involved in the research.

Research that has meaningful coherence effectively does what it sets out to do. It will tell a clear story. That story may include paradox and contradiction, mess and disturbance. Nevertheless, it will bring together theory, literature, data and analysis in an interconnected and comprehensible way.

These criteria are not an unarguable rubric to which every qualitative researcher must adhere. Indeed there are times when they will conflict in practice. For example, you may have a delightfully resonant vignette, but be unable to use it because it would identify the participant concerned; participants may not be willing or able to be involved beyond data provision; and all the diligence in the world can’t guarantee a significant contribution. So, as always, researchers need to exercise their powers of thought, creativity, and improvisation in the service of good quality research, and use the criteria flexibly, as guidelines rather than rules. However, what these criteria do offer is a very helpful framework for assessing the likely quality of research at the design stage, and the actual quality of research on completion.

Next week I will post a case study demonstrating how these criteria can be used.

Working With Indigenous Literature

Indigenous methods booksI have been working with Indigenous literature on research methods and ethics for over a year now: reading, annotating, thinking, re-reading, and writing from this literature alongside Euro-Western literature on the same topics. During this process I have been reflecting on how I should, and how I do, work with Indigenous literature.

Working well with any literature requires careful and thorough reading and reflection. Beyond that, though, it seems to me there are some specific requirements for working with Indigenous literature.

To begin with, readers need to understand that although the current body of literature is small, it represents a much longer research tradition than that represented by Euro-Western methods/ethics literature. Indigenous researchers around the world, through their oral history, can trace their research tradition back over 40,000 years (Passingan 2013:361; Steere 2013:388-391). By contrast, in Euro-Western terms, anything over 6,000 years old is pre-history. However, there is also a translation problem. Although the first language of some Indigenous people these days may be English, the community and tribal languages of Indigenous peoples worldwide are very different. The complex concepts people develop when talking and thinking about research and its context may not translate directly between a community or tribal language and English, or vice versa. Even when words can be translated, the ideas those words describe may not be fully translatable to people with different world views. For example, some Indigenous peoples regard their ancestors as involved in the research they conduct. This is an unusual concept for most Euro-Western researchers and it is unlikely that we can understand fully what that involvement means to the Indigenous peoples concerned.

Then readers need an awareness that translating an oral tradition into writing is a big intellectual and emotional challenge. There is evidence within the Indigenous research literature that some of its authors are ambivalent about writing. For example, Shawn Wilson states that ‘A problem with writing down stories is that it makes it very difficult to change them as we gain new learning and insights.’ (2008:22) He solves this problem in his own work by writing, not in the Euro-Western linear style of introduction, methods, findings etc, but in ‘a more cyclical pattern that introduces ideas or themes, then returns to them at intervals with different levels of understanding’ (ibid:42). He also tries to write relationally, in accordance with Indigenous ethical principles, i.e. to build a relationship between himself as author, his readers, and the ideas he puts forward (ibid:6). With this aim in view, he uses a literary device of addressing his three children in sections of the book. He chose to do this because, unlike his unknown readers, he has an existing relationship with his children which enables him to use a more personal ‘voice’ in writing for them (ibid:9). Wilson’s hope is that these sections, alongside the more conventionally written ones, will help his readers to develop their own relationships with the ideas and themes in his book.

Wilson and others (e.g. Lambert 2014) also include stories from other named people in their books. This requires a change in citation practices, as simply citing the name of the person on the cover of the book does not adequately acknowledge the source of information. I have solved this problem by citing this way: ‘Lewis in Wilson 2008:110’ or ‘Leslie Camel in Lambert 2014:167’. (I have yet to find out whether my publisher regards this as OK or not, though I hope they will.)

It seems to me that Indigenous literature should be worked with in its own ethical terms. Indigenous research ethical principles are not the same as Euro-Western research ethical principles. While they are described differently by different writers, there are four key ethical principles that recur in the Indigenous literature: respect, accountability, relationality and reciprocity. For me, Indigenous literature should be treated with respect, such as by not dismissing concepts that seem unusual or even alien to me, but by giving them full consideration. If I can’t embrace a particular concept myself, at the very least I need to acknowledge and respect its significance for some Indigenous peoples.

Accountability comes with putting my name to my writing on these topics, making my views available on my blog and (next year) in a book, and giving anyone the right to comment or reply, whether or not they agree with me. Relationality involves careful reading and thinking and use of the literature, trying to understand the different works and authors in relation to me, and me in relation to them, as well as the relationships between the works and authors themselves. This is very much a work in progress and I hope one day to extend it into actual relationship with individual Indigenous researchers, though whether or not any Indigenous researcher would find value in that remains to be seen.

Reciprocity is the principle I find the most difficult when working only with literature. I hope I can achieve this by writing from the Indigenous literature in a way that its authors will value, and in a way that will enable other Euro-Western researchers and scholars to find and learn from that literature as I myself have done. I have been very concerned that working with the literature could be extractive, i.e. taking ideas and concepts and using them for my benefit alone. It is true that the books I write help me to earn a living, but I am not getting rich from them. Nobody pays me while I write, and my income from writing in 2016-17 was around £1,500 (I don’t have the exact figure yet, but will post it when I do). Also, I don’t write niche academic monographs, but textbooks that have proven to be of interest to postgraduate students and academics around the world. I found comfort in direct requests from some Indigenous writers to include their literature in Euro-Western works (Kovach 2009:13,25; Graham Smith in Kovach 2009:88-89; Chilisa 2012:56), and I hope that raising the profile of this literature within the Euro-Western research world will prove to be a positive act. If it backfires in some unforeseen way, on Indigenous individuals and communities who have already borne too much pain, all I will be able to do is acknowledge and own my responsibility for my part in that process, to remain accountable, learn, and do what I can to right any wrongs.

This post is a starting point rather than a conclusion. I have figured all this out myself, so I may be on the wrong track. If you are working with, or have an interest in, this body of literature, please contribute your views in the comments.

Self-Care And Privilege

France summer 2009 2 for Helen 137I wrote a post in December 2014 about the importance of self-care. I’m taking my own advice again and going on holiday for a fortnight. That means this blog will be quiet too, as the pre-holiday frenzy of client work and writing deadlines did not allow for preparing and scheduling posts.

There’s a lot of talk about self-care for researchers, the self-employed, and people who do scholarly work. Quite right, too; it’s important in these 24/7 professions. But it’s easy for us to forget that self-care is also a privilege.

I’ve been thinking a lot about privilege, partly because I’m writing about ethics and partly because of world affairs. There’s a lot of talk about white privilege, for good reasons, but I don’t think that’s the whole story. For me, privilege is intersectional, just like disadvantage. So some of my privilege is about being white, but I have other forms of privilege too. I have enough time, money, health, and support from others – and I need all of those privileges to be able to take this holiday. There are white people with no privilege beyond their whiteness; they still have privilege, but it does not help them much.

I have a lot of privilege other than my whiteness. For example, I haven’t had to ask anyone for permission to take time off, and I’m not using up a portion of precious annual leave. In fact, I can take as many holidays as I like, in theory, though in practice I have to prioritise working for a living. But that’s going OK because I also have enough money, not only to pay for a holiday (albeit a cheap one), but also to spend a couple of weeks not earning. I thought about my health privilege this morning, as I spoke to a family member, younger than me, who hasn’t had a holiday in years for health reasons. And I have the privilege of a loving partner, family, and friends, who support me.

There are even more forms of privilege in play here. For example, I will be thinking about my freedom-of-movement privilege as I travel through Calais, where so many refugees are suffering. And of course white privilege counts too: I will not experience racism at any point on this trip, and nobody is likely to harass or try to kill me because of the colour of my skin.

I have been reading articles about people living very different lives, dealing with monstrous injustice around the world. I have been reading about people in my own country who can’t afford to eat. Part of me thinks that, with all these horrors going on, I shouldn’t even have a holiday. But a bigger part of me refuses to feel guilty for taking this opportunity. I will, though, be aware of how very lucky I am.