Why Peer Reviewing Is More Difficult These Days

Posted by Helen Kara

I have been a peer reviewer of journal articles for the last eight years. I documented my first peer review, in late 2014, on this blog. Peer reviewing has never seemed easy to me – and I don’t think it should. Reviewing original work by other scholars is bound to be intellectually and emotionally demanding. But I feel as if peer reviewing has become more difficult, even over the comparatively short time I have been involved. There are several reasons for this, and I will focus on three of them here: hoaxes, malpractice and complexity.

Academic hoaxes pre-date my reviewing experience. In 2005, three US-based doctoral students in computer science, Jeremy Stribling, Max Krohn and Dan Aguayo, created SCIgen. SCIgen is a computer program which can generate whole computer science journal articles including graphs, figures and citations, that look credible but are in fact nonsensical. A lot of articles generated by SCIgen have been accepted by, and published in, academic journals, despite the use of peer reviewers.

And such hoaxes are not limited to computer science. In 2017–18, three UK-based scholars, James Lindsay, Helen Pluckrose and Peter Boghossian, wrote 20 fake articles using social science jargon. They were able to get several of these articles published in academic journals, even though some of them promoted morally questionable acts. The aim of these three scholars was apparently to highlight what they saw as poor quality work in some areas of the social sciences. However, I am not sure this intended end justifies the questionable means of duping reviewers and editors into publishing bogus research.

Sadly, though, it seems that academic journals are regularly duped into publishing bogus research by researchers themselves. Retraction Watch, based in the US, has been keeping track of retracted journal articles for the last 12 years. Some articles are retracted because their authors made honest mistakes. But the Retraction Watch database lists a lot of other reasons for retraction, including falsification or fabrication of data, and falsification, fabrication or manipulation of images or results. And the numbers are staggering. At the time of writing, there are over 1,500 articles listed on the database as retracted due to the falsification and/or fabrication of data, and over 1,000 due to the manipulation of images. Also, the database only includes those articles in which fabrication, falsification or manipulation have been detected and reported. By its own admission, Retraction Watch is biased towards the life sciences, so problematic journal articles in other sectors will be even less visible.

A bunch of people make it their business to find and publicise these problematic articles. One even does it under her own name: Elisabeth Bik. Others use pseudonyms such as Clare Francis, Smut Clyde, Cheshire, and TigerBB8.

Bik specialises in identifying manipulated images, and has found through empirical research that their prevalence is increasing. However, Bik has a particular talent for pattern recognition. Of course it is useful to know that images may be manipulated, and Bik regularly shares examples on social media and elsewhere which can help others understand what to look for. But even so, spotting manipulated images can be difficult for the average, harassed, unpaid peer reviewer. And catching fabricated or falsified images, data or results may be almost impossible without inside information. Most journal articles have strict word limits which can work against them here. These restrictions mean researchers are used to some aspects of their processes receiving a cursory mention at best, and this can enable cheating to pass undetected.

When reviewing goes wrong, consequences can be disastrous. The link is to a recent controversy about a published article promoting a morally questionable act. I am not using any of its keywords in this article. I think there are some particularly interesting aspects of this case. It is not the first article to be published that features morally questionable acts. I have read the article; it is well written, and I can see how a peer reviewer could regard it as worthy of publication – as its own peer reviewers did. The problem, for me, lay in the background of the author who promotes morally questionable acts outside of academia. He may have written this article in the hope that publication would lend legitimacy to his actions. Even if he did not, publication might be perceived to confer such legitimacy, which could cause reputational damage to the publisher and the university concerned.

So, the article you are reviewing may be a hoax, and/or may contain data, images, and/or results that have been manipulated, fabricated or falsified, in ways that are difficult or impossible to detect, and/or may have been written by someone with a dodgy agenda. But that’s not all. Academic work – and, indeed, the world around us – is becoming more complex. More research is transdisciplinary, pushes methodological boundaries, is multi-lingual, and so on. The process of peer review was devised when people worked in neat, tidy, single disciplines and fields. In that landscape people could act as experts on other people’s work in its entirety. These days that is not so easy. Topics such as sustainability, the climate crisis, and food security transcend disciplines and methods. This means that nobody, really, is an expert any more, so peer review is effectively obsolete. Yet it is still being used.

This means we need not only peer review before publication, but also after publication. Luckily there is a tool for this: PubPeer, a website where you can comment on published journal articles, anonymously if you wish. This enables researchers with inside information to whistleblow without risking the loss of their jobs. Also, you can use PubPeer to check articles you are intending to cite, to make sure nobody has raised any concerns about the work you want to use. At the moment PubPeer focuses mostly on laboratory and clinical research, but there is also (not surprisingly) some computer science. In fact PubPeer can be used for any published journal article as long as the article has a recognisable ID such as a DOI. Also, there is a PubPeer browser plugin which enables PubPeer comments to be visible on other websites besides PubPeer itself.

This blog and the videos on my YouTube channel are funded by my beloved Patrons. Patrons receive exclusive content and various rewards, depending on their level of support, such as access to my special private Patreon-only blog posts, bi-monthly Q&A sessions on Zoom, free e-book downloads and signed copies of my books. Patrons can also suggest topics for my blogs and videos. If you want to support me by becoming a Patron click here. Whilst ongoing support would be fantastic you can make a one-time donation instead, through the PayPal button on this blog, if that works better for you. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

How To Give Feedback On Academic Writing – Twelve Top Tips

Posted by Helen Kara

feedback people A recent discussion on Facebook reminded me that I’ve written about how to deal with feedback from reviewers, but I haven’t written about how to give feedback to peers and colleagues. There is an art to this which I have learned, paradoxically, from receiving feedback, which taught me what helps and what does not help.

Feedback is a fairly neutral word but what we’re actually dealing with is criticism. Some people call it ‘critique’ to make it sound better but it’s still criticism. Criticism is not neutral and so it has lots of emotion attached.

In the last decade I joined a closed online short story writing group of around a dozen fiction writers. We all knew each other online through blogging and wanted to improve our writing. The idea was that we would each write and share a story once a fortnight. The stories were posted anonymously by one of the group – we took turns – and the others would give feedback. To begin with we only gave positive feedback until one of us pointed out that we weren’t going to get very far that way. We were a bit scared about being more critical, but gradually our feedback became more robust, with honesty about the elements of each story that didn’t work for us and why, as well as praise for the parts that did and suggestions for how to overcome weaknesses. We built up a lot of trust in that group and it helped us to give better feedback and so become better writers.

This experience taught me that trust is important to effective feedback. In the group we built trust over time. If you’re writing an anonymous peer review, you need to create trust all at once.

Another thing that is important is blending praise where possible, or at least advice, with your criticism. I had a review for the typescript of my last book which was entirely critical. Essentially, it said the book was rubbish and should never be published. The reviewer is entitled to their opinion, and I have been a writer for far too long to be upset by critical feedback, but the problem was that the review gave me no help at all. There was nothing in it which I could use to improve my writing. (Luckily I had two other reviewers at that stage who took a more balanced approach and did give me constructive criticism, advice, and some praise.)

So, from all my years of experience of receiving and giving feedback on writing in several genres, here are my twelve top tips for giving good quality feedback that others will trust.

Be honest in all the feedback you give.
Read the piece you’re giving feedback on carefully, thoroughly, at least twice.
While you read, make notes of thoughts that occur to you. As a minimum, these should include: aspects of the work you think are good; where you think there is room for improvement; anything you don’t understand; references the author might find helpful.
Be sure to praise the good points in the author’s work. This helps to build trust and also lets the author know what they can relax about.
Be open about anything you don’t understand. Doing this worries some people because they think they may look stupid, particularly if they’re giving feedback to a peer or colleague rather than writing an anonymous review. But it’s really helpful feedback for writers because it may be that they haven’t written clearly enough.
Give a straightforward assessment of areas where you think there is room for improvement.
Tell the author how you think they can improve their work. This is crucial. If you’re only saying where improvement is needed, you’re only doing half the job.
Where relevant, suggest references the author has missed.
If you think extra references would be helpful but nothing specific springs to mind, have a quick look on a website such as Google Scholar or the Directory of Open Access Journals and see if you can find something to point the author towards.
Don’t worry if you can only offer a certain amount of help because of the limits to your own knowledge. It’s fine to say, for example, that a quick online search suggests there is more relevant literature in the area of X; you’re not certain because X lies outside your own areas of interest but you think it would be worth the author taking a look.
Acknowledge the author’s emotions. For example, after giving quite critical feedback, you might say something like, “I realise that implementing my suggestions will involve a fair amount of extra work and this may seem discouraging. I hope you won’t be put off because I do think you have a solid basis here and you are evidently capable of producing an excellent piece of writing.” (Though remember #1 above and don’t say this if it’s not true.)
Be polite throughout, even if your review is anonymous. Anonymity is not an excuse for rudeness.

If there’s anything I’ve missed, please add it in the comments.

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $12 per month. If you think four of my blog posts are worth more than $12 – you can help! Ongoing support would be fantastic but you can also support for a single month if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to give financial support at this time, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

How To Deal With Reviewers’ Comments

Posted by Helen Kara

editing text Your first set of reviewers’ comments lands in your inbox. Your heart begins to race. Will your work be accepted or rejected? Will they love it or hate it? Can you bear to open the email?

These may be reviews for a journal article, book proposal, or book typescript. In each case the process is the same. First you need to read the comments and give yourself time to react. Whether it’s the exultation of an acceptance, the despair of a rejection, or the mixed feelings that come with requests for revisions, you need time to process your emotional response before you do anything else. Whoop, cry, eat chocolate, do whatever you need to do.

Because of negativity bias, negative comments – even when constructively phrased – have more impact on most people than positive comments. We need to work to counteract this bias. So, unless you’ve received very favourable comments and you want to revel in their glory, I recommend waiting at least 24 hours before you read the comments again. This can help you to take a more balanced view, which is useful because if it’s a rejection or revisions, you’ll need to see how your work can benefit from the reviewers’ input before you send it off again. This can be quite a challenge, especially if the reviewers have different views of your work and how it can be improved. Your journal or commissioning editor may offer some guidance and if so you should take that into account. But sometimes they leave it all to you.

My solution to this is to treat the reviewers’ comments as data and go into analysis mode. I create a table with one column for the comments and another for each reviewer. Then I enter each substantive comment into the first column and put a mark in the other columns for each reviewer who has made a similar point. This helps me to pick up the instances where reviewers are effectively saying the same thing, though perhaps in very different ways. It also helps me to see at a glance which comments have been made by all or some reviewers, and which only by one of the reviewers.

I have said before on this blog that reviewers’ comments come in three categories: the no-brainer (act on this), the no-thanks (don’t act on this), and the oh-wait (probably act on this, though not necessarily in the way the reviewer suggests). So my next job is to sort comments into these categories.

If a comment has been made by more than one reviewer I will take it more seriously. That doesn’t mean I’ll definitely implement it, but I am more likely to do so. If a comment has been made by all reviewers I would need a very, very good reason not to implement it. If a comment has only been made by one reviewer, that in itself might be one reason I could decide not to implement it, though I would also expect to give at least one other reason.

Once I have sorted the comments into their categories, I will list them by category in the first column of another table with two further columns: a brief note of what I plan to do in response to each of the no-brainers and the oh-waits, and a brief note of what I plan to write in the cover letter against each comment from all three categories. This is useful because I can dip into it when I have a spare half-hour or so, and find a job or two to do to get me closer to the finish line.

It is important to be polite in your response to reviewers’ comments, even if you think they’re the biggest load of old rubbish you’ve seen since your last visit to the municipal tip. Some reviewers’ suggestions seem to be based more on what they would have written than on what you actually have written and this can be quite annoying at times. When you come across a suggestion you really don’t want to implement, there are some tactful ways to say so, such as:

“This is an excellent suggestion though unfortunately beyond the scope of this particular project.”

“I can see how this suggestion would improve my work but sadly I am unable to incorporate it within the allocated word count.”

“This is a really interesting idea. I have considered it carefully and concluded that it doesn’t quite fit with the thrust of my current article/book, but it will influence my thinking for future projects.”

Remember you are the author and, as such, you have authority. While authors do need reviewers’ input (at least, when it’s constructive), and your work should benefit from intelligent use of their feedback, you don’t have to do everything a reviewer says. Also, a rejection is only a rejection from this journal or publisher. It doesn’t mean your work is worthless; sometimes it’s only because they already have plans to publish something that is similar in some way. This post should help you make the best use you can of reviewers’ comments. That will produce the greatest benefit to your work and career, and is also a way to respect and honour the time and care (most) people put into writing reviews.

Thoughts On Writing Book Chapters

Posted by Helen Kara

Sage handbook of QRE I have written two chapters for edited collections, both on qualitative research ethics. The first was for a book called Qualitative Ethics in Practice, edited by Martin Tolich and published in 2016 by the late lamented Left Coast Press. I said ‘yes’ to that one straight away because it was the first time I’d been asked. Writing the chapter was an interesting and enjoyable exercise but economically pointless. I got a free copy of the book, but I could have bought the paperback for £24.99 from Amazon or, no doubt, for less elsewhere. (I recommend using the book price comparison site Bookbutler, though it doesn’t index all sellers; I don’t see Wordery on that site, and Wordery often have good discounts as well as free shipping worldwide. eBay is also worth checking for discounted new copies; as an author myself I am not advocating buying secondhand books). Given that the chapter took me at least a week to write and edit, an affordable paperback is poor recompense. Also, book chapters don’t carry the academic kudos of journal articles, so they don’t do much for my reputation with universities.

When I was a doctoral student, I loved a good edited collection for offering a range of viewpoints and arguments within a single book. As a reader, I still do, when it’s well done. That suggests I should contribute to such collections. Yet there is so little recompense.

I thought about this carefully. On the morning of 5 January 2016 I decided it wasn’t worth the effort, and made a belated New Year’s resolution that I wouldn’t write another book chapter. On the afternoon of 5 January 2016 I got an email from Ron Iphofen and Martin Tolich asking me to write another book chapter, for the Sage Handbook of Qualitative Research Ethics which they were beginning to co-edit. Ron and Martin are colleagues with whom I get on well, and that makes it harder to say ‘no’. I did say that I could not take on a chapter requiring primary research or any kind of extensive review of literature with which I wasn’t already familiar. (Well done, past Helen!) After some discussion we found an angle that worked, as it would cover an area where I already had some knowledge that I needed to develop, and it also suited the editors.

I got my copy of the book this week. It’s almost 600 pages, 35 chapters, and retails at £120 on Amazon. On one hand, that still represents woeful recompense for several days of work. On the other hand, £120 is way beyond my budget for buying any book, even one as extensive as this book. And I’m very glad to have a copy.

So I’m rethinking the whole book chapter thing again. Now I think I am more likely to say ‘yes’ if the book in question will be big and expensive and useful for my work. I also think I’ll keep to my decision not to write chapters that need primary research or extensive new reading. Some new reading is inevitable, and that’s OK, but essentially I’m only going to write chapters that I can actually write straight from my desk.

Another thing I have learned about writing book chapters is to ask the editors for the book proposal, so I can see where my chapter fits, and not duplicate work others may be doing in their chapters. It doesn’t seem to be common practice for editors to give the book proposal to potential contributors (I’m not basing this solely on my own experience, I’ve heard the same from other academic writers) though I expect some do. If you’re asked to write a book chapter and the editor doesn’t give you the book proposal, ask for it before you decide. It can give you a much clearer idea of what you’re contributing to.

As with all academic writing intended for publication, book chapters are likely to be peer reviewed individually, and the typescript of the whole book is also likely to be reviewed. (The proposal will have been reviewed, too, before being accepted by the publisher.) So be prepared for edits, proofs etc to come your way. You may also be asked to review a chapter by another author, as sometimes book editors and commissioning editors get around the difficulty in finding reviewers by having their chapter authors review other chapters. Overall, there will be more work than just the writing.

I’m currently reviewing the typescript of a book which is reminding me how much I like a good edited collection. The book’s theme is strong and consistent, and the variation in the chapters is fascinating, in terms of both their content and how authors are addressing the topic. This offers a particular type of richness that no single or co-authored book can achieve. So I’m content with my decision, now, not to say a blanket ‘yes’ or ‘no’ to requests for book chapters. I will say ‘yes’ to writing chapters that will benefit me in the process and the outcome, as well as benefiting editors, publishers, and readers.

Is Peer Review Bad For Your Mental Health?

Posted by Helen Kara

peer review people I am currently waiting for peer reviews of two books I’ve worked on: one sole-authored, one co-authored. We don’t talk much about the experience of waiting for reviews, and it’s not something that appears to have been researched. Yet it’s something everyone doing academic work has to go through and it may be bad for our mental health.

I’m finding it particularly difficult at the moment because a lot rides on these reviews. The sole-authored book, which is on a contentious topic, has already had one set of reviews. Reviewer 1 was utterly damning, saying ‘I couldn’t find anything to praise’. Luckily, reviewers 2 and 3 were more measured, offering both praise and constructive criticism, and their input helped me to revise and strengthen the typescript. However, in the process, my editor and I realised that we needed further reviews from people with a particular kind of specialist knowledge. My editor approached around 10 potential reviewers, but only one agreed to do the job. So I’m gibbering – what if that person agrees with Reviewer 1?

The co-authored book is in a contentious format. My co-authors and I decided that I would be the person to liaise with publishers, as I have form in this process. The last time I liaised with academic publishers for a co-authored book was in the early 2000s, and I’d forgotten how heavily responsible it makes me feel. Fortunately, I’ve found a publisher that is interested and has sent the book out for reviews, to two professional academics and two students. This is great – and terrifying – but at least there are four reviewers. Even so, what if they ALL think it’s rubbish?

In many ways I love the peer review system. I welcome feedback on my writing, and I’m not at all averse to constructive criticism. I am by no means arrogant enough to think I can write a good book without input from others. Yet peer review, as a process, is fraught with uncertainty. Comments may not be constructive, or may not come at all. They may be positive, or negative, or in between, or a mixture.

There is a body of research which demonstrates that uncertainty has a detrimental effect on mental health. Luckily for me, my mental health is fairly robust right now, so I can use this period of uncertainty as an opportunity to build further resilience. But what about those who aren’t so fortunate? We’re hearing a lot about the mental health crisis in higher education, but nobody seems to be talking about the potential contribution of the peer review system to this crisis. Given the evidence of links between uncertainty and mental health, it seems likely that there may be a relationship here.

I’m not arguing that we should tear down the peer review system and replace it with something completely different. Time spent waiting for reviews also has a positive effect, in that it creates necessary distance between the author and their work, meaning sensible revisions are easier to make. But I do think we need to be aware of the effects of uncertainty and take steps to reduce its impact on us. Here are four ideas.

Aspects of life fall into three categories: those you can control, those you may be able to influence, and those you can neither control nor influence. Spend most of your energies on the first, some on the second, and none on the third. So I will spend my energies on hard work and good fun, and with any luck I won’t have much energy left for fruitless worry about the outcome of the peer reviews.

Plan for different outcomes. Plans for positive reviews are easy, plans for negative reviews more challenging. For me, the worst-case scenario is that the publisher decides not to publish after all, which would mean – for either book – several years of work down the pan. However, that is unlikely, and if it does happen I/we can revise and submit again elsewhere.

Acknowledge how you’re feeling. Writing this blog post is one way for me to acknowledge my own difficult feelings about this waiting period. In professional UK society the culture is not to talk about feelings much, if at all; if anyone asks how you are, the standard answers to give include ‘I’m fine’, or (with an eyeroll) ‘snowed under’. It’s as if we’re not allowed to give a real answer to the question. Yet suppressing our emotions is also bad for our mental health, so let’s talk about the difficulty of waiting, being in limbo, for unpredictable peer reviews.

Practise self-care. All the usual stuff: eat sensibly, take exercise, get enough sleep – or, if you can’t sleep, rest your body quietly in a dark room and try to still your mind. There are some good video soundtracks and podcasts online to help you sleep. Work can be part of self-care when it’s work you enjoy and you don’t do too much. Spending time with loved ones is definitely part of self-care.

The peer review system can also be hard on reviewers, such as by asking more of people who are already too busy, and offering only intangible rewards. Saying ‘yes’ to a review request adds an extra burden of work, saying ‘no’ comes with an extra burden of guilt. Some people deal with this by deciding how many reviews they will undertake, such as 12 in a year, or three per draft article or book they themselves submit. That’s a great example of focusing on what you can control.

If you’re waiting for reviews yourself, the wait will be over, sooner or later. I hope you will be able to use the advice in this post to help make the process a little easier – as I intend to do myself. I wish you luck.