Why I Love Reviewer 2

pencils and heartFeedback can feel like a very mixed blessing at times. Positive feedback is a delight to receive, while even the most constructive criticism can come as a crushing blow. Writers are particularly susceptible to this, especially novice writers who haven’t yet learned to separate critique of their writing from critique of themselves. I often meet doctoral students who are very reluctant to show their work to their supervisors, fearing criticism because they’re worried that it’s not very good. If it’s a first draft, of course it’s not very good, and a second draft will also contain problems that have to be fixed. Supervisors need to see this work so they can give feedback, which should include information about:

  1. what you’re doing well,
  2. what needs improvement, and
  3. how you can make those improvements.

If any of these elements is missing, ask them to include it in future feedback.

More experienced writers can also struggle with feedback. “Reviewer 2”, referring to an anonymous peer reviewer of an academic journal article, has become a standing joke on social media.

Roses are red, violets are blue, why are you so loathsome, Reviewer 2?

Even when you are really experienced, with a thesis or dissertation, several journal articles, book chapters, and even books to your name, feedback can pack an emotional punch. When you receive feedback (which should be in writing), read it through and give yourself time for emotional as well as cognitive digestion. If anything in the feedback annoys or upsets you, apply self-care: chocolate, a hug from a loved one, walking outdoors, meditation, gardening, exercise – whatever works for you. Then, when you’re ready, read it again and find the key messages.

Here’s some of the feedback from my book proposals:

  • The synopsis is quite antagonistic
  • This will provide insufficient information to be useful
  • The thrust of the book remains unclear
  • Chapters 1 and 8 seem to be somewhat repetitive
  • It is a bit thin and not complex enough to add anything new
  • The proposal covers a wide terrain and is unfocussed
  • I think this would be an excellent edited book… the author would benefit from the input… it’s a very broad aim otherwise and may not succeed
  • Far greater clarity is needed
  • The book will not make a very original contribution
  • The writing style is stiff

Luckily there was also a fair amount of positive feedback. Positive feedback is great: it provides much-needed encouragement, and lets you know what you can relax about. But it’s the “Reviewer 2” type comments that really help you improve.

There are three sensible ways to respond to constructive criticism. First, the no-brainer. Chapters 1 and 8 seem repetitive? That’s useful and specific, so I would definitely check those two chapters against each other and remove any unnecessary repetition.

Second, the no-thanks. An edited collection rather than a sole-authored book? I thought that could potentially make the aim even broader, with a bunch of authors jockeying for position. Luckily, my editor agreed.

Third, the oh-wait. The book will not make a very original contribution? I was sure it would, but what this comment told me, crucially, was that I had not communicated the originality of the contribution well enough in my proposal. It is so important to remember that reviewers can be wrong – though if they are, the fault probably lies in your writing. (Not always. Some good scholars are poor reviewers, especially those who are unable to distinguish the piece you are writing from the piece they would write if they addressed the same topic. But usually.) So when you are considering feedback on your writing, don’t always take it at face value. Think about it in the context of your work as a whole, and make a decision. You should certainly take notice if more than one reviewer says similar things. Another reviewer on the same proposal says more clarity is needed. The two comments, together, tell me I have not been clear enough about the contribution I think the book can and will make. That is very useful information because I need the book’s contribution to be perfectly clear by the time of publication, so I and my publisher can communicate it to potential readers. More work evidently needed.

This decision-making can be difficult, and sometimes a second opinion is helpful. Reviewers, too, can be unclear. If you don’t understand what a reviewer is trying to say, it’s tempting to jump to the conclusion that they’re cleverer than you and reach for the despair. However, it may well be that they haven’t articulated their point effectively, which unfortunately makes your job harder rather than easier. Sometimes you can go back to them for clarification; it’s fine to do this, even if you have to go through an intermediary such as a journal editor. But it is sensible to check with someone else first, to make sure it’s not a comparatively straightforward point that you’re just missing for some reason.

I always welcome feedback on my writing. I can’t write a book, or anything else for that matter, without feedback from a range of people. Critical feedback doesn’t discourage me, or at least not for long. The only time I’ve had a journal article rejected is when I wrote one for a client; I told them I would need feedback on a draft from a suitably experienced person, and they said someone from their organisation would provide this, but when the time came they said they couldn’t and I should just send in my draft. I was sure that wouldn’t work, and indeed it didn’t. After that I was able to persuade them to find me someone who could offer feedback; their input was very helpful, and the article was accepted by our second choice of journal.

I understand that some people struggle with feedback. I understand why some people struggle with feedback. But honestly, if you’re one of those people, and you want to succeed as a writer, you need to find a way over, through, past, or around that struggle. I hope this post will help, and that you will learn to love Reviewer 2 as much as I do.

The Importance Of Creative Research Methods

me presenting at CRMSS17Last Thursday, Friday and Saturday I was privileged to facilitate the inaugural Creative Research Methods summer school run by Keele University‘s Cultural Animation and Social Innovation Centre (CASIC) working with the New Vic Theatre in nearby Newcastle-under-Lyme. Around 40 people came, travelling from America and South Africa, Sweden and Poland, no doubt other countries I’ve forgotten, and all around the UK.

On the first two mornings we were lucky enough to get to work in the theatre’s auditorium, a wonderful space with plenty of room to move around and interact with people in all sorts of ways. On the first day we used pipecleaners to model journeys both literal and metaphorical, and on the second day we explored issues of power in research using Open Space Technology.

For the first two afternoons, we crossed the car park to the theatre’s Workspace rehearsal room, another great space – with a balcony! On the first afternoon we learned about cultural animation, used buttons to create community maps, then added frames and artefacts to help us come up with research questions. Then we devised and performed creative group presentations – that was so much fun! On the second afternoon we mapped pathways through participation in universities, using flip chart paper, coloured Post-It notes and pens, pipecleaners and tape – by now the creative juices were really flowing.

On the third day we were at the beautiful Keele campus, where (as it was a Saturday) we could use some of the university’s technology facilities: the KAVE for virtual reality and gaming, the Claus Moser studio for soundscapes, and the Turing Lab to make digital circuits. In the afternoon we focused on creative academic writing, hearing about ethnography as advocacy for the animals who are often invisible in social research, and geopoetics, before doing a geopoetics exercise.

We crammed in a great deal, yet there was so much else we could have included. Perhaps the richest part of the summer school was its discussions: between any two people, or a group, or all of us together. I was delighted and astonished by the calibre of the students: an enormously intelligent, creative, dynamic bunch; it was an honour to spend three days in their company.

I love to teach creative research methods, and I’m looking forward to my next gig this Friday at LSE for the National Centre for Research Methods (fully booked I’m afraid). I find a lot of my teaching involves giving people permission to work creatively – or perhaps enabling them to give themselves permission – and advising people on how to convince supervisors and ethics committees that it is legitimate to take a creative approach to research. There is a long hard fight ahead to convince people in certain quarters that useful knowledge exists beyond the bounds of academic convention. In this fight, we are on the same side as Indigenous researchers around the world who find their methodologies are sidelined or ridiculed by the academy. Anishnabe researcher Kathy Absolon, in conversation with Plains Cree and Salteaux researcher Margaret Kovach, said this:

If you go on a water walk or quest, that is your methodology. I was reflecting when you were talking about yours [methodology]. If I said I am doing my PhD and my methodology is my dreams, and I am going to go on a fast every year, and after that fast I had somebody come and visit me and talk to me about my fast and take [teachings] with them. I wouldn’t propose that because I wouldn’t want that to [be] measured. I know that is Indigenous methodologies, but I wouldn’t propose it as a methodology within a mainstream setting because I don’t want them to have the power to say that that’s not research. But it is. (Absolon in Kovach 2009:152-3)

There is a parallel here with creative research in the Euro-Western paradigm, where supervisors, ethics committees, journal editors and reviewers, and others have the power to say ‘this is not research’ to people who know perfectly well that their textile art, ice-skating, or poetry, is indeed research. Patricia Leavy has written eloquently of ‘the ache of false separation’ that some people feel when required to keep their art separate from their research work (2010:240).

Some people have said to me that one reason I can write the books I write is that I’m not an academic. As an independent researcher, I have much less power than many academics, in many ways. But I do have the power to say ‘this is research’, and to collect the evidence that this is research, and put it in a scholarly book, so that other people can cite that work, which helps to convince doubting/frightened/threatened supervisors and others. And I will stand with Indigenous researchers, though their methods are not my methods, because I recognise that knowledge comes from more places and in more ways in this complex and beautiful world than those I can access myself.

Still it feels lonely sometimes. So having the opportunity to spend three days with a group of lively-minded people, who are not only open to this but engaging with it, excited by it, and pushing its boundaries in fascinating ways, was an absolute delight.

Indigenous Research Methods: A Reading List

Indigenous methods booksLast week I wrote about challenging the dominance of English in writing for research and academia. That theme is also relevant to this post, though here it’s more about challenging Euro-Western epistemologies and methods than the English language itself. Over the last year I have built a personal library of books about, or relevant to, my investigation of Indigenous research methods and ethics. The point of this, for me, is to bring these methods into my scholarship, alongside creative and conventional methods, as appropriate. The point is not to become an ‘expert’ on Indigenous research; for a white British person, that is not, should not be, an option. At the start of this work, I worried about being extractive, but I found comfort in the words of Margaret Kovach, an Indigenous researcher from Saskatchewan in Canada, who encourages non-Indigenous scholars to help make space for Indigenous methodologies and assess their value on their own terms. This is what I am trying to do.

For those who are new to this topic, ‘Indigenous’ denotes the native peoples of colonised lands, such as Aboriginal Australians or Inuit Alaskans, while ‘indigenous’ denotes the native peoples of non-colonised lands. So I am an indigenous Brit who will never be an Indigenous researcher. Some people described as Indigenous are unhappy with the term because they feel that it makes them seem like one homogeneous group, whereas in fact there is tremendous diversity. For example, there are hundreds of tribal and language groupings in Australia alone. However, as it is the term most commonly used in the literature, I’m sticking with it for now.

The first book is the foundational Decolonizing Methodologies by Linda Tuhiwai Smith, a Maori researcher from New Zealand. In fact I bought the first edition of this soon after it came out in 1999, the year I began my MSc in Social Research Methods. The second edition came out in 2012. This book shows how research was used as a tool of imperialism to help subjugate colonised peoples through, among other things, complete disregard for Indigenous knowledges and Indigenous peoples’ own research methods. It highlights the value of these knowledges and methods, and calls for research to be linked explicitly with social justice.

Shawn Wilson is an Opaskwayak Cree researcher from Canada who has also lived and worked with Indigenous peoples in Alaska and Australia, as well as spending time with Indigenous peoples in New Zealand, Morocco, and elsewhere. His book, Research Is Ceremony: Indigenous Research Methods (2008), is based on his doctoral research and describes a paradigm shared by Indigenous researchers in Canada and Australia. It’s not easy to get hold of; I tracked down a Canadian bookseller who seems to have bought up the last available copies, and I fear it may be going out of print, which would be a great shame as it is readable and insightful.

Margaret Kovach is a Plains Cree and Salteaux researcher from Canada whose Indigenous Methodologies: Characteristics, Conversations, and Contexts came out in 2009. Her book covers epistemologies, methods, and ethics. It is a work of considerable scholarship that is also accessible and full of wisdom.

Bagele Chilisa is a Professor at the University of Botswana. Her book Indigenous Research Methodologies (2012) gives an uncompromising and international account of some of the theories, epistemologies, ontologies and methods used by Indigenous researchers. While no book on this subject could be completely comprehensive, Chilisa makes a good job of showing the diversity, as well as some of the commonalities, of Indigenous methodology.

Donna Mertens from the US, Fiona Cram from New Zealand, and Bagele Chilisa have edited a collection called Indigenous Pathways into Social Research: Voices of a New Generation (2013). They have contributions from Indigenous researchers from all around the world: Vanuatu, Mexico, Cameroon, Hawai’i, Alaska, Papua New Guinea, and many other countries. These are fascinating accounts, highlighting personal, political, and ethical challenges, and how they have been overcome. They also say a lot about Indigenous methodologies around the world.

Also in 2013, Maggie Walter, a trawlwoolway researcher from Tasmania, and Chris Andersen, a Métis researcher from Canada, brought out Indigenous Statistics: A Quantitative Research Methodology. This book demonstrates the pervasiveness of Euro-Western thought in the construction of statistical research, using national censuses for ilustration. It offers a framework for Indigenous quantitative research, nayri kati or ‘good numbers’, which places an Indigenous standpoint at the centre. There is a short video online of Maggie Walter talking about Indigenous quantitative research.

Lori Lambert is a Mi’kmaq researcher from north-eastern Canada who has also worked with Indigenous peoples from Montana, US; northern Manitoba, Canada; and Queensland, Australia. Her book, Research for Indigenous Survival: Indigenous Research Methodologies in the Behavioral Sciences, was published in 2014. To the best of my knowledge, this is the first book to position Indigenous methods within a Euro-Western disciplinary category. Like other Canadian writers, such as Wilson and Kovach (above), Lambert includes the voices of people she has worked with alongside her own in her narrative.

Another essential text, though not specifically about research methods, is Southern Theory by Australian academic Raewyn Connell (2009). This book is subtitled ‘The global dynamics of knowledge in social science’ and in my view is essential reading for anyone engaging with social theory. During my MSc, I was taught social theory as the preserve of dead white men, and I am sure this is still being taught in many Euro-Western universities today. Connell’s book gives the lie to this approach.

This list is not exhaustive; it is just my personal library. One limitation is that I can’t afford expensive books. While I was writing this blog post, I had a message from my friend and colleague Roxanne Persaud, alerting me to Susan Strega and Leslie Brown’s edited collection Research as Resistance: Revisiting Critical, Indigenous, and Anti-Oppressive Practices (2nd edn 2015). I would love to read this book, but even the paperback is over £60 which puts it out of my reach.

These books are not comfortable reads for Euro-Western scholars, but they are hugely important. We need to know how research has been, and is, misused by Euro-Western cultures in order to learn how to use it better. Indigenous scholars are extraordinarily generous in their assessment of the potential value of Euro-Western methodologies, even those methodologies that have been instrumental in stealing their lands and their cultures and traumatising generations of their peoples. Yet most Euro-Western researchers either ignore Indigenous research entirely, or conclude that Indigenous peoples must have picked up a few tricks from the colonisers. I’m not sure which is worse. Indigenous research methods pre-date Euro-Western research methods by tens of thousands of years, and there is a great deal that Euro-Western researchers can learn from these approaches.

Challenging The Dominance Of English

languagesIn a thought-provoking blog post, Naomi Barnes of Brisbane, Australia, recently asked what other white people were doing to break down the barriers built by whiteness. This is a very good question. One thing we can do is to challenge the dominance of English.

Language is not neutral in research or education. English is the dominant language of both, worldwide, as a direct result of colonialism. English is dominant even though it ranks only third in the world: more of the world’s people speak Mandarin Chinese, or Spanish. Studying for a PhD (or equivalent), or writing an academic journal article, is demanding enough when you can do it in your native language. Every year, around the world, millions of people have to study and write in English when that is not their native language, which makes already difficult work much more difficult. People like me, who are born into an English-speaking country, are unbelievably lucky and have a massive head-start. A lot of us, I think, don’t realise how lucky we are.

Professor Bagele Chilisa of Botswana, in her excellent book from 2012 on Indigenous Research Methodologies, calls this the ‘hierarchy of language’. (The English version of the search engine I use, duckduckgo, has never heard of her book, which rather proves her point.) The hierarchy of language comes with a range of ethical implications for native English speakers, and I will outline three of the main ones here.

First, we need to understand that there is not just one form of English, there are many: from Bangalore to Boston, from London to Lagos, from Sydney to Sao Paulo. This means we should not assume that someone’s ideas have less worth because their spoken English is heavily accented, or formulated differently from our own, or their written English is not entirely fluent.

‘Language-ism’ is embedded in structures such as academia and publishing. People who write non-standard English, regardless of the quality of the content, are less likely to have their work formally published in academic journals – or, at least, not the journals usually indexed by Google Scholar or the Directory of Open Access Journals. This is one of the ‘barriers built by whiteness’ referred to by Naomi Barnes. As a result, work in non-standard English is harder to find, so it is less likely to be used, shared, or cited. Yet some of these researchers are doing excellent work which is well worth exploring.

This is the second ethical point: we need to try harder to find, and use, work by non-native English speakers. Those of us who can read other languages have a head-start here. (Many non-English speaking countries teach languages, including English, to children throughout their schooling. In England in the 1970s, when I was at school, learning other languages was mostly optional – I spent just three years learning elementary French and have only needed to use it, since then, when actually in France. Even there many people speak better English than my French. This is another indication of the dominance of English.) But whether or not you can read other languages, you need to know where to look for research from beyond the countries where English is dominant. Here are some ‘starters for 10’ thanks to Andy Nobes of INASP on Twitter, in conversation with Raul Pacheco-Vega, Pat Thomson and Jo VanEvery:

African Journals Online

Bangladesh Journals Online

Central American Journals Online

Journals from Latin America, the Caribbean, Spain and Portugal

Latindex (Latin America, the Caribbean, Spain and Portugal – Spanish only)

Mongolia Journals Online

Nepal Journals Online

Philippine Journals Online

Scientific Electronic Library Online (Latin America, Spain, Portugal and South Africa)

Sri Lanka Journals Online

Many of these are supported by the research, knowledge and development charity INASP through its Journals Online project. Most have an English option on their website and some, if not all, articles available in English. Much of the content is openly accessible.

The third ethical point is to look at this the other way around. If we write in English, we should do all we can to get our work translated into other majority languages. There are 23 languages in the world that are each spoken as a first language by over 50 million people. The top 10 are: Chinese, Spanish, English, Arabic, Hindi, Bengali, Portuguese, Russian, Japanese and Lahnda (a Pakistani language). Translation brings its own ethical problems, as there is not always a straightforwardly equivalent word for an idea or a concept, so translating from one language to another can involve some creativity and interpretation. However, a careful translation between any two majority languages will make your work available to many more scholars. In particular, translations from English help to reduce its dominance.

So there are three ways for white people (and native English speakers of colour) to challenge the dominance of English and so help to break down some of the barriers built by whiteness. If you can think of other ways to do this, please add them in the comments.

The Variety Of Indie Research Work

varietyOne of the things I love about being an independent researcher is the sheer variety of projects I work on and tasks I might do in a day. Yesterday, I was only in the office for the afternoon, yet I worked on at least seven different things. Here’s what I did.

First, I checked Twitter, and found a tweet with a link to a blog post I wrote about an event that is part of a project I’m working on with and for the forensic science community. This is a new departure for me, in that I haven’t worked with forensic scientists before, though the work itself is straightforward. I’m supporting a small group of people with research to identify the best way to create a repository for good quality student research data, and it’s surprisingly interesting. So I retweeted the tweet.

Second, I dealt with the morning’s emails. The arrival of a purchase order I’d been waiting for weeks to receive – hurrah! I formulated the invoice and sent it off to the client. Then some correspondence about the creative research methods summer school I’m facilitating at Keele in early July – just three weeks away now, so the planning is hotting up (and there are still some places left if you’d like to join us – it’ll be informative and fun). The most interesting email was a blog post from Naomi Barnes, an Australian education scholar who is considering what it means to be a white educator in the Australian school system. This chimes with the work I am doing on my next book, so I leave a comment and tweet the link.

While on Twitter, I got side-tracked by a tweet announcing #AuthorsForGrenfell, an initiative set up by authors for authors to donate items for auction to raise funds for the Red Cross London Fire Relief Fund to help survivors of the Grenfell Tower fire. I’d been wanting to help: my father is a Londoner, I have always had family in London, I lived in London myself from 1982-1997, and one member of my family is working in the tower right now to recover bodies. So it feels very close to home. But I’m not in a position to give lots of money, so I was delighted to find this option which I hope will enable me to raise more money than I could give myself. I have offered one copy of each of my books plus a Skype consultation with each one. My items aren’t yet up on the site, but I hope they will be soon because bidding is open already. If you’re one of my wealthy readers, please go over there and make a bid!

Then I spent some time researching aftercare for data. Yes, indeed there is such a thing. So far I’ve come up with two ways to take care of your data after your project is finished: secure storage and open publication. They are of course diametrically opposed, and which you choose depends on the nature of your data. Open publication is the ethical choice in most cases, enabling your data to be reused and cited, increasing your visibility as a researcher, and reducing the overall burden on potential research participants. In some cases, though, personal or commercial sensitivities will require secure storage of data. There may be other ways to take care of data after the end of a project, and I’ll be on the lookout for those as I work on my next book.

By now it was 6 pm so I did a last trawl of the emails, and found one from Sage Publishing with a link to a Dropbox folder containing 20 research methods case studies for me to review. They publish these cases online as part of their Methodspace website. I like this work: it’s flexible enough to fit around other commitments and, like other kinds of review, it tests my knowledge of research methods while also helping me to stay up to date. Best of all, unlike other kinds of review, Sage pay for my expertise. So I downloaded all the documents, checked and signed the contract, and emailed it back with a ‘thank you’. By then it was 6.30 pm and time to go home.

As the old saying goes, variety is the spice of life. I certainly like the flavour it gives to my work. Some days I work on a single project all day; those days are fun too. Yesterday I worked in my own office, today I’m out at meetings locally, tomorrow I’m off to London. It’s always ‘all change’ and I wouldn’t have it any other way.

Let’s Talk About Research Misconduct

detective-152085__340Research misconduct is on the rise, certainly within hard science subjects, quite possibly elsewhere. Researchers around the world are inventing data, falsifying findings, and plagiarising the work of others. Part of this is due to the pressure on some researchers to publish their findings in academic journals. There is also career-related pressure on researchers to conduct accurate polls, produce statistically significant results, and get answers to questions, among other things. Some clients, managers, funders and publishers have a low tolerance for findings that chime with common sense or the familiar conclusion of ‘more research is needed’. They may expect researchers to produce interesting or novel findings that will direct action or support change.

Publishers are working to counteract misconduct in a variety of ways. Plagiarism detection software is now routinely used by most big publishers. Also, journal articles can be retracted (i.e. de-published) and this is on the increase, most commonly as a result of fraud. However, the effectiveness of retraction is questionable. The US organisation Retraction Watch has a ‘leaderboard’ of researchers with the most retracted papers, some of whom have had more papers retracted than you or I will ever write, which suggests that retraction of a paper – even for fraud – does not necessarily discredit a researcher or prevent them from working.

Some research misconduct can have devastating effects on people, organisations, and professions. People may lose their jobs, be stripped of prizes or honours, and be prosecuted in criminal courts. Organisations lose money, such as the cost of wasted research, disciplinary hearings, and recruitment to fill vacancies left by fraudulent researchers. And whole professions can suffer, as misconduct slows progress based on research. For example, in 2012 the Journal of Medical Ethics published a study showing that thousands of patients had been treated on the basis of research published in papers that were subsequently retracted. Retraction Watch shows that some papers receive hundreds of citations even after they have been retracted, which suggests that retraction may not be communicated effectively.

Yet even the potentially devastating consequences of misconduct are clearly not much of a deterrent – and in many cases may not occur at all. Let’s examine a case in more detail. Hwang Woo-Suk is a researcher from South Korea. In the early 2000s he was widely regarded as an eminent scientist. Then in 2006 he was found to have faked much of his research, and he admitted fraud. Hwang’s funding was withdrawn, criminal charges were laid against him, and in 2009 he received a suspended prison sentence. Yet he continued to work as a researcher (albeit in a different specialism) and to contribute to publications as a named author.

Closer to home, a survey of over 2,700 medical researchers published by the British Medical Journal in 2012 found that one in seven had ‘witnessed colleagues intentionally altering or fabricating data during their research or for the purposes of publication’. Given the pressures on researchers, perhaps this is not surprising – though it is deeply shocking.

The examples given in this article are from hard science rather than social research. Evidence of misconduct in social research is hard to find, so it would be tempting to conclude that it happens less and perhaps that social researchers are somehow more ethical and virtuous than other researchers. I feel very wary about making such assumptions. It is also possible that social research is less open about misconduct than other related disciplines, or that it’s easier to get away with misconduct in social research.

So what is the answer? Ethics books, seminars, conferences etc frequently exhort individual researchers to think and act ethically, but I’m not sure this provides sufficient safeguards. Should we watch each other, as well as ourselves? Maybe we should, at least up to a point. Working collaboratively can be a useful guard against unethical practice – but many researchers work alone or unsupervised. I don’t think formal ethical approval is much help here, either; it is certainly no safeguard against falsifying findings or plagiarism. Perhaps all we can do at present is to maintain awareness of the potential for, and dangers of, misconduct.

A version of this article was originally published in ‘Research Matters’, the quarterly newsletter for members of the UK and Ireland Social Research Association.

Creative Methods for Evaluation: A Frustration

frustrationEvaluation is a particular type of applied research designed to assess the value of a service, intervention, policy or other such phenomenon. This is relevant to all of us as it forms the basis for many decisions about public service provision. Despite being applied research, evaluation also has a significant academic profile, with dedicated journals, many books, and university departments with professors of evaluation in countries around the world.

There are a range of types of, and approaches to, evaluation research. They all have some things in common: they start with the desired outcomes of the service, intervention etc; formulate indicators that would show those outcomes had been met; then collect data in line with those indicators and analyse it to identify the extent to which the outcomes have been met. So, for example, if a community service aims to reduce loneliness, they might decide that one indicator could be a reduction of reports of loneliness to community-based doctors and nurses, then work with health colleagues to collect information from health records before and after the provision of the service to show whether there was any difference. Evaluators also write recommendations for ways to improve the service, intervention etc. The intention is that these recommendations are implemented, then later reviewed in another cycle of evaluation research.

The basics of general research practice also apply to evaluation: plan thoroughly, collect and analyse data, produce written and other outcomes, and publish your findings. From time to time I teach a course called ‘Creative Research Methods for Evaluation’, usually as part of the UK and Ireland Social Research Association’s open training programme. All sorts of people come on this course: central and local Government researchers, charity researchers, health researchers, researchers in private practice, research funders – a real mix, which makes it great fun.

I know quite a bit about creative methods; after all, I wrote a book on the subject. I tell my students about arts-based methods, research using technology, mixed methods, and transformative research frameworks. We talk about when these methods are appropriate to use, and how they can work side-by-side with more established methods. I give them lots of examples of creative methods in use.

And here is the huge frustration. While I have plenty of examples of creative methods in practice, very few come from evaluation research. I have some examples from my own practice, though only as verbal stories because the written and other outputs are subject to client confidentiality. This is a big problem with evaluation research: because it is applied, i.e. often conducted by and for individual organisations, it is rarely published beyond its immediate audience. When it is published, it is often simply uploaded to a web page and so disappears into the depths of the internet. And if it is both published and findable, it is not likely to include the use of creative methods.

There are many examples of perfectly competent evaluations using well-established methods. However, evaluators today are working on complex projects and benefit from having more methodological options at their fingertips. I know my course helps, because former students have told me so, but during the course someone always asks why I’m not using examples from evaluation research. (Even though I explain this problem at the start!) I wish I could use such examples; I’m sure they’re out there; but even though I have searched, and asked, and searched again, I can’t find them. So this is by way of an appeal: do you know of any good resources that showcase creative methods in evaluation research? By ‘good resources’ I mean well written outputs or short engaging videos (3-4 minutes at most) that are not too basic, as my students are generally quite experienced. If you have anything to suggest, please let me know in the comments.

Broken Academic Promises

broken promiseLast week I wrote a post begging established academics to keep their promises (or not make them in the first place). This started some interesting conversations with academics and independent scholars across Twitter and the blogosphere.

Peter Tennant, a healthcare researcher at Leeds University, responded on Twitter with sympathetic outrage.

Naomi Barnes, an independent researcher working towards a career in academia in Brisbane, Australia, shared her own experience.

Sarina Kilham, an academic from Sydney, Australia, made a good point about it being important for professionals to respect the value of each others’ time.

UK independent researcher Anne Cummins commented on the original blog post, simply saying: ‘This has happened to me too.’

Deborah Brian, an indie researcher from Brisbane, Australia, had a real horror story to tell.

An anonymous Australian academic, who tweets from a locked account as @snarkyphd, said: “I hire 20+ casuals /yr. Am clear when I can’t make promises, but lose great people to those who promise the world + don’t follow through.” And then: “What I observe is tendency to make promises to twice as many people as realistic so you don’t end up short staffed later on. So frustrating.” (I can’t embed her tweets in this post because her Twitter account is locked, but I do have her consent to share her words here.)

Naomi Barnes, in conversation with academic career guide Jo VanEvery, came back with another example:

Claudia Gillberg, a researcher in the UK and Sweden, broadened the debate further from her experience of living and working with disability.

Around this time I caught up with last week’s post from the Research Whisperer, written by Dani Barrington, on playing the academic game. She identifies ‘compromising my own values and the wellbeing of others’ as a common part of the ‘academic game’, which seems to parallel these conversations.

The last comment on my own post, last week, was from Jenni Brooks who is a senior lecturer in sociology in the UK. She says,

Thanks for writing this Kara. It’s important (but I confess slightly uncomfortable) reading for me, as someone in a permanent academic job in a university.

I realise that I’m often enthusiastic when I meet someone doing good work, and yes, if I’d love to work with someone, I will say. I hope that’s always accompanied by a qualification if there’s a chance things may never happen, but it’s quite possible that it isn’t, and that I too have promised things I can’t keep (I do always reply to my emails though, I promise!)

Being relatively new to a teaching (as opposed to research only) post, I’m still coming to terms with the ‘students above all’ mentality that means other things get put on the backburner sometimes, even if I’d rather they weren’t. For example, I was meant to be writing a funding bid with a friend in another university recently which, because of marking and other deadlines, just hasn’t happened. Neither of our jobs depends on it, so we’ll just do it later.

I wonder whether this very thing that you talk about does make me wary of working with people outside of the ‘university system’ though, whether that’s researchers, charitable organisations, or ‘service users’ (or whatever other appropriate term). I’m aware of the imbalance in funding/job security, and yes, it scares me if I feel in some way responsible for making sure someone else is paid – especially if I’m not entirely sure of how the system works. I’m also aware that this can (and has, in the past) made me seem ‘cagey’, and in some way like I’m trying to ‘protect university resources’, or ‘not share’ in some way.

I’m not entirely sure what my point is, but just wanted to say how thought-provoking your post was. I can see I need to do a lot of thinking about how to be a good ‘academic citizen’ (is that a thing?)

I think it is a ‘thing’. Evidently, so do many others. To be fair, independent researchers and scholars also need to be good citizens and colleagues. But as with all power imbalances, it is those who have the power who need to wield it carefully.

A Plea To Established Academics: Please Keep Your Promises

crossed-fingers-363478__340The university sector in the UK is in a shaky state at present. Several universities across England, Wales and Scotland have each announced over 100 redundancies, sometimes with the closure of a school or campus, and it seems likely that others will follow suit. Brexit is the reason most people give, though there may be other reasons too. The uncertainty is unsettling for mid- to late career academics, and very worrying indeed for early career researchers and doctoral researchers with academic ambitions. It’s also worrying for independent researchers, like myself, who work with academia.

There is a four-panel comic strip at PhD Comics called Things You Can Do In Academia That Would Get You Fired In The Real World. The ‘things’ listed are:

  • Abandon personal grooming
  • Be a jerk at other people’s presentations
  • Not reply to emails
  • Sit around and do nothing all day (quote: “it’s called writing”).

I would add a fifth ‘thing’:

  • Continually make promises you don’t keep.

The income of independent researchers depends on established academics keeping their promises. Given the number of imminent redundancies forecast in the UK higher education sector, there are going to be more independent researchers looking for work in the months and years to come. This post is a plea to established academics not to make promises, to colleagues in more precarious positions, that you can’t or won’t keep.

I have had many, many promises made to me by established academics, and I would estimate that over half are not kept. These kinds of promises have come through social media, at conferences, during my workshops, even at one-to-one face-to-face meetings specifically set up to explore how we can work together. I’ve never been an academic, so I don’t know for sure, but I would guess this is common, perhaps normal, within academia. When both of you have a secure salary, it probably doesn’t matter so much; perhaps it’s just a small irritation or disappointment when plans don’t come to fruition – or maybe sometimes it’s a relief. However, it’s a whole different deal when one of you depends on the other to follow through.

Unkept promises that come to mind include:

  1. “You must come and teach my students. Send me your CV and we’ll fix a date.” I send my CV, and never hear from the Professor again.
  2. “I’d love to write with you. Let’s do a journal article.” This is someone I would love to write with, too, so I say yes – then nothing happens.
  3. “I want to allocate some time in this funding bid for you to work with us on the project, is that OK with you?” Yes, I say, and agree to give input on the draft bid – then nothing happens.
  4. “I want you to come and speak at my event, and I have a budget to pay for your time and your expenses.” Great! Yes please! Then, guess what? Nothing happens.
  5. “Thanks so much for your workshop/seminar/class. The feedback was great. We’ll have to get you back next year.” Sometimes this does happen; mostly it doesn’t.
  6. “I’d be happy to give feedback on your draft article.” That promise is kept, and then, “I think you need to include a section on X,” which is the specialism of the academic in question. I agree. The academic offers to write the section and become second author; I accept with pleasure. I never hear anything further AND I ALREADY WORKED ON THAT ARTICLE FOR A WHOLE YEAR.
  7. “I’d like to give you an honorary position as Visiting Professor, which will mean I’ll be able to pay you to work with our students.” Sounds great, yes please. Only, er, it doesn’t happen.

What’s worse is that, while numbers 6 and 7 were one-offs, numbers 1-5 in the list above have each happened several times. I spent a couple of decades working as an employee in the private, statutory, and voluntary sectors, and I can tell you this doesn’t happen there. It didn’t happen in my independent research work, either, in the 2000s, when my clients were local and central government departments and charitable organisations. I’m not saying every professional promise made outside of academia is kept, but unkept promises are usually acknowledged, and the promisee kept up to date with reasons for the change of plan. At times I’ve had to chase by email asking for an update; I have always received a full reply outside academia, while within academia I have rarely received a reply at all. To be fair, some academics do communicate about their unkept promises, just as some manage to write, answer their emails, be courteous at presentations, and maintain a good level of personal hygiene. But too many do not.

Established academics, with a steadily increasing number of your colleagues in ever more precarious positions, you really need to be aware of this. Each of the promises above made my day and raised hope and excitement in my heart. Then, as the days, weeks, months went by with no further communication, the creeping feelings of disappointment, frustration, and disillusionment were sometimes hard to bear. And I’m an old hand at this. I think it’s going to be much more difficult for younger people who are desperately working so very hard to try to make a career in, or with, academia. So I beg you, established academics, please think carefully about the promises you make, and stop making promises you can’t or won’t keep.

Writer’s Block Debunked

writers blockI don’t believe in writer’s block. I think it’s an umbrella term for a whole bunch of problems, each of which has a solution. For sure, writing is difficult, and it is perfectly possible to get stuck. That happens to me all the time. But a complete block? I don’t think so. Seems to me ‘writer’s block’ is a lazy catch-all term which conveniently shifts responsibility from the writer to the ‘block’. That’s certainly not going to help.

Here are six of the things I think may be really going on.

Fear of failure. Writing for submission or publication can be terrifying. You’re working towards putting your work out in the world where it – and to some extent you – will be judged. What if people don’t like your writing? What if they don’t like YOU? This kind of fear can be paralysing.

Fear of success. Success equals change, and change is frightening, because it’s a leap into the unknown. This can be equally paralysing.

Boredom. This is a higher risk for long-term projects, or writing that has been imposed on you – say, by your manager, or a departmental imperative – that you don’t want to do.

Perfectionism. Novice writers sometimes think that writing is (or should be, or is for some people) as easy as reading. It isn’t. To write, you have to be willing to write badly first, and then make it good at the editing stage, sometimes through many revisions.

Running out of steam with a particular avenue or genre. Minette Walters is a successful author of psychological thrillers who published 12 books between 1992 and 2007, then didn’t publish another book for 10 years. She had become bored with writing psychological thrillers, but her publisher wanted her to carry on, presumably because they sold well. Now she has found a new publisher and is bringing out a historical novel. This can apply for non-fiction and scholarly writers, too, particularly if you’ve been writing in a specific area for some time and long to change tack.

Self-sabotage. If you say, think, believe that you want to write, but you’re not writing, then you are in some way sabotaging your own desires. This is a common human trait and probably links back to fear of failure, or perhaps fear of success.

Here are five potential solutions.

Freewrite. I love the technique of freewriting, and so do the doctoral students I teach it to. Here’s what you do. Set yourself a prompt, which must be in the first person and active voice, such as:

What I want to say is…

In this chapter, I want to argue…

I am writing this [thesis/dissertation/article/report/etc] because…

Then write for five minutes without stopping or correcting your work. This is only for your eyes so it doesn’t matter how scrawly or mis-spelled it may be. If you hesitate during the five minutes, write the prompt again, more than once if you need to, until it leads you somewhere else. Then see what you’ve got. You may well have a new insight or a phrase or sentence that you can use in your project. More importantly, though, you’ll have a load of words on a page, which – in five minutes flat – gives the lie to ‘writer’s block’.

Set small targets. Some people prefer word targets; others prefer time targets. Either is fine. Choose a target that feels easily achievable: perhaps somewhere between 100-500 words, or 10-30 minutes. Then set a frequency: once a day is good to begin with, or twice if you can manage that. And stick to it.

Switch projects. Working on more than one project is great because if you get bored, or stuck, you can move to another. I didn’t have a single idea for a blog post this morning, so I worked on a short story for a while till I came up with the idea of writing about writer’s block.

Keep a daily journal. Writing about your life, or some aspect of your life, for your eyes only, is a great way to convince your subconscious that you can write. This can be structured, or unstructured, as you prefer. For example, you might want to keep a journal of your dreams, or a ‘resilience journal’ where you write down three things you are grateful for and three things you did well each day, or a ‘reflective journal’ where you record what you have learned that day. Or you might just want to write whatever you feel like writing.

Go for counselling. If fear of failure, or fear of success, are really getting in the way – and, for some people, they can – then find a counsellor or therapist who can help you work on this.

If you have other solutions to share, please add them in the comment box.