Why Peer Reviewing Is More Difficult These Days

Posted by Helen Kara

I have been a peer reviewer of journal articles for the last eight years. I documented my first peer review, in late 2014, on this blog. Peer reviewing has never seemed easy to me – and I don’t think it should. Reviewing original work by other scholars is bound to be intellectually and emotionally demanding. But I feel as if peer reviewing has become more difficult, even over the comparatively short time I have been involved. There are several reasons for this, and I will focus on three of them here: hoaxes, malpractice and complexity.

Academic hoaxes pre-date my reviewing experience. In 2005, three US-based doctoral students in computer science, Jeremy Stribling, Max Krohn and Dan Aguayo, created SCIgen. SCIgen is a computer program which can generate whole computer science journal articles including graphs, figures and citations, that look credible but are in fact nonsensical. A lot of articles generated by SCIgen have been accepted by, and published in, academic journals, despite the use of peer reviewers.

And such hoaxes are not limited to computer science. In 2017–18, three UK-based scholars, James Lindsay, Helen Pluckrose and Peter Boghossian, wrote 20 fake articles using social science jargon. They were able to get several of these articles published in academic journals, even though some of them promoted morally questionable acts. The aim of these three scholars was apparently to highlight what they saw as poor quality work in some areas of the social sciences. However, I am not sure this intended end justifies the questionable means of duping reviewers and editors into publishing bogus research.

Sadly, though, it seems that academic journals are regularly duped into publishing bogus research by researchers themselves. Retraction Watch, based in the US, has been keeping track of retracted journal articles for the last 12 years. Some articles are retracted because their authors made honest mistakes. But the Retraction Watch database lists a lot of other reasons for retraction, including falsification or fabrication of data, and falsification, fabrication or manipulation of images or results. And the numbers are staggering. At the time of writing, there are over 1,500 articles listed on the database as retracted due to the falsification and/or fabrication of data, and over 1,000 due to the manipulation of images. Also, the database only includes those articles in which fabrication, falsification or manipulation have been detected and reported. By its own admission, Retraction Watch is biased towards the life sciences, so problematic journal articles in other sectors will be even less visible.

A bunch of people make it their business to find and publicise these problematic articles. One even does it under her own name: Elisabeth Bik. Others use pseudonyms such as Clare Francis, Smut Clyde, Cheshire, and TigerBB8.

Bik specialises in identifying manipulated images, and has found through empirical research that their prevalence is increasing. However, Bik has a particular talent for pattern recognition. Of course it is useful to know that images may be manipulated, and Bik regularly shares examples on social media and elsewhere which can help others understand what to look for. But even so, spotting manipulated images can be difficult for the average, harassed, unpaid peer reviewer. And catching fabricated or falsified images, data or results may be almost impossible without inside information. Most journal articles have strict word limits which can work against them here. These restrictions mean researchers are used to some aspects of their processes receiving a cursory mention at best, and this can enable cheating to pass undetected.

When reviewing goes wrong, consequences can be disastrous. The link is to a recent controversy about a published article promoting a morally questionable act. I am not using any of its keywords in this article. I think there are some particularly interesting aspects of this case. It is not the first article to be published that features morally questionable acts. I have read the article; it is well written, and I can see how a peer reviewer could regard it as worthy of publication – as its own peer reviewers did. The problem, for me, lay in the background of the author who promotes morally questionable acts outside of academia. He may have written this article in the hope that publication would lend legitimacy to his actions. Even if he did not, publication might be perceived to confer such legitimacy, which could cause reputational damage to the publisher and the university concerned.

So, the article you are reviewing may be a hoax, and/or may contain data, images, and/or results that have been manipulated, fabricated or falsified, in ways that are difficult or impossible to detect, and/or may have been written by someone with a dodgy agenda. But that’s not all. Academic work – and, indeed, the world around us – is becoming more complex. More research is transdisciplinary, pushes methodological boundaries, is multi-lingual, and so on. The process of peer review was devised when people worked in neat, tidy, single disciplines and fields. In that landscape people could act as experts on other people’s work in its entirety. These days that is not so easy. Topics such as sustainability, the climate crisis, and food security transcend disciplines and methods. This means that nobody, really, is an expert any more, so peer review is effectively obsolete. Yet it is still being used.

This means we need not only peer review before publication, but also after publication. Luckily there is a tool for this: PubPeer, a website where you can comment on published journal articles, anonymously if you wish. This enables researchers with inside information to whistleblow without risking the loss of their jobs. Also, you can use PubPeer to check articles you are intending to cite, to make sure nobody has raised any concerns about the work you want to use. At the moment PubPeer focuses mostly on laboratory and clinical research, but there is also (not surprisingly) some computer science. In fact PubPeer can be used for any published journal article as long as the article has a recognisable ID such as a DOI. Also, there is a PubPeer browser plugin which enables PubPeer comments to be visible on other websites besides PubPeer itself.

This blog and the videos on my YouTube channel are funded by my beloved Patrons. Patrons receive exclusive content and various rewards, depending on their level of support, such as access to my special private Patreon-only blog posts, bi-monthly Q&A sessions on Zoom, free e-book downloads and signed copies of my books. Patrons can also suggest topics for my blogs and videos. If you want to support me by becoming a Patron click here. Whilst ongoing support would be fantastic you can make a one-time donation instead, through the PayPal button on this blog, if that works better for you. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

Knowing When To Stop

Posted by Helen Kara

stop Sometimes it’s hard to know when to stop. That could be when you’re still having fun and you don’t want to stop even though it’s after midnight and you’ve got to be in work at 9. In my early 20s I could get away with that. In my mid-50s? No chance. The dark sides of not knowing when to stop are dependency and addiction. Then there are the mental ‘ought’s and ‘should’s. I ought to finish reading this book, that I’m not enjoying at all, because the author took so much trouble in its writing. I should keep working on this collaborative piece even though my collaborator hasn’t answered my emails in months.

There’s an art to knowing when to stop. My mother, who is prone to outbreaks of wisdom, explained to me the point of stopping while you’re still having fun. Because what’s the alternative? Keep going till you’re not having fun any more? If you do that, you’re unlikely to want to do whatever-it-was again. Whereas if you stop while you’re still having fun, you keep the magic.

The ‘ought’s and ‘should’s can bog off. There are so many books (and journal articles, and – ahem – blog posts, and so on) that if you’re not enjoying one, why take the time to read it to the end? You won’t have time in your whole life to read all the books (journal articles, blog posts etc) that you do enjoy. So blow it out. Read the last page/paragraph if you need to satisfy your curiosity, then toss it and find something that suits you better.

And as for collaborations that have gone belly-up… that can be hard, when you’ve put in a lot of work and you’re not far from the finish line. But recognising when you need to quit is an important survival skill because it protects you from throwing good time after bad.

There’s another way this can work, too, which is not so much knowing when to stop as recognising that you have stopped. This has happened to me with my New Year’s resolution (I know! July! Not bad, eh?). My resolution was to review a book a week; i.e. an academic book, and to publicise this and encourage others to join in. I said from the start that it didn’t actually have to be a book a week, and I followed my own guidance; I reviewed 14 books between 1 January and 7 June, 12 on Wordery/Amazon and two for the LSE blogs. I haven’t reviewed an academic book in the last couple of months, though I’ve read quite a few. I will continue to review academic and other books but I’m not going to plug it as a ‘thing’ any more.

This is partly because hardly anyone joined in. A few people said it was a great idea, and one or two did write reviews, but it was evidently an idea whose time has not come, or has passed, or will never exist. Conversely, the monthly creative methods chat that I started in June has taken off rather well. And of course the point of all these things is not only to be the thing in itself, but also to raise my profile. Sounds cynical, in a way – yet I’m running a business and I have books and skills to sell. That doesn’t mean I’m trying to sell them to everyone all the time; that would clearly be unrealistic. I aim to create initiatives which will be of value to people in themselves, because I think that’s the best way to do marketing. Not to shout GIVE ME WORK AND BUY MY BOOKS AND BE MY PATRON but to generate resources and opportunities for people, which may lead to some of those people choosing to put some work or money my way. Or not – there’s no obligation and I like it that way. But the return on investment for these initiatives is low. For example, there have been over 5,000 downloads of Starting Your PhD: What You Need To Know and around 25 reviews worldwide, or one review for every 200 downloads. So evidently it’s sensible to invest time in the initiatives that increase my visibility rather than those that don’t, no matter how close they may be to my heart.

So bye bye, review a book a week. It was nice knowing you. And hello, #CRMethodsChat. You’re ace, and you happen on the second Tuesday of every month. Long may that continue.

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $34 per month. If you think a day of my time is worth more than $34 – you can help! Ongoing support would be fantastic but you can also make a one-time donation through the PayPal button on this blog if that works better for you. Support from Patrons and donors also enables me to keep this blog ad-free. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

New Year’s Resolution: Review A Book A Week

Posted by Helen Kara

books Happy New Year, lovely blog readers! I hope 2019 is full of happiness for each and every one of you.

My New Year’s resolution this year is to spread a little happiness by reviewing an academic book each week. Academic books, even those that are widely read and cited, rarely receive public reviews. Yet public reviews online are the most useful tools to help potential readers decide whether or not to read a book. People are using reviews more and more: to find ways to meet their needs for everything from holiday accommodation to plumbers. I’m ashamed to admit I wrote more reviews on TripAdvisor last year than I did on Amazon.

That’s a sticking point, of course. Some people are ideologically opposed to using Amazon because of the company’s exploitative employment policies and avoidance of tax. Another option is Goodreads – though (little-known fact coming up) Goodreads are owned by Amazon, which I only found out as I looked them up online for information to share here. Yet Amazon and Goodreads are the most useful sites from a potential reader’s viewpoint because they are where most book reviews are posted.

For an ethical alternative, Wordery are independent and ship worldwide for free. There aren’t many book reviews on Wordery as yet; the website is more interested in promoting reviews for its business than in encouraging book reviews. This may be because it is a newish business, founded in 2012. But there is space to write reviews on Wordery.

Of course I could also review on social media, and sharing information about books that way is helpful. However it’s not as permanent, and doesn’t enable comparison of different viewpoints in the same way, as reviewing on a website. Reviewing on blogs is good, especially the more widely read blogs, but writing a whole blog post is much more demanding for the reviewer. A review on Amazon/Goodreads/Wordery need only be a few sentences long.

If you’re not sure how to review a book, here are two top tips. First, give an honest opinion of what you liked or disliked, or found useful/not useful, with reasons. Reviews that say ‘This book is pointless’ or ‘This book is marvellous’, without explaining why, are not helpful. An example from my own approach: I always deduct a star for an academic book with an inadequate index or no index at all, because for me this reduces the usefulness of the book. When I’m working I need to be able to navigate swiftly around a book’s contents and a good professional index is an essential aid. But this is a personal requirement, so explaining why I’ve taken away a star is helpful for potential readers who may have different requirements. For example, some people only ever read a book once and make careful notes as they read which they use for reference later. For those people, an index is much less important.

Second, say what kind of people you think will find the book useful. That could be people at a particular stage of education, or with specific interests or needs, or studying/working in a certain discipline or field. You can do more if you wish, but if you do those two things, you will have written a review which could help others decide whether to spend time and/or money on the book in question.

I’ve written my first review for this year on amazon.co.uk and copied it to Wordery. (I also tried to copy it to amazon.com, as I have done in the past, but found I’m ineligible because I haven’t spent $50 there in the last year.) The book I chose to review was Indigenous Research Methodologies by Bagele Chilisa which I have mentioned before on this blog. This illustrates another important point: reviewing a book a week doesn’t mean reading a book a week. I will review books I read during the year, and I will also review a selection of the books from my shelves that I haven’t yet reviewed. I plan to prioritise books by women, queer people, scholars with disabilities, Indigenous writers, and others who have to contend with oppression.

As an author myself, it would be disingenuous of me not to declare that reviews help authors too. Bagele Chilisa’s book has (at the time of writing) 1109 citations on Google Scholar, yet only one review on Amazon UK and three on Goodreads. While citations are great if you’re in academia, public reviews increase visibility for authors far more than citations. I have never understood why academic readers don’t take a few minutes to write public reviews like readers of other types of books. Though I’m guilty too… but that is going to change! The minute I publish this blog post I’m going to write my second review for this year.

You can join in if you too would like to spread a little happiness. All you need to do is take five minutes to write a short public review of an academic book. Perhaps a book you think should be more widely known, or that you would not recommend (don’t forget to say why), or that would help readers in a particular category. Even if you only review a book a month – or even a book a year – that will help potential readers, and authors too. I’ll be using the hashtags #reviewabook and #reviewabookaweek to talk about this on social media. Hope to see you there!

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $12 per month. If you think 4-5 of my blog posts is worth more than $12 – you can help! Ongoing support would be fantastic but you can also support for a single month if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

How To Give Feedback On Academic Writing – Twelve Top Tips

Posted by Helen Kara

feedback people A recent discussion on Facebook reminded me that I’ve written about how to deal with feedback from reviewers, but I haven’t written about how to give feedback to peers and colleagues. There is an art to this which I have learned, paradoxically, from receiving feedback, which taught me what helps and what does not help.

Feedback is a fairly neutral word but what we’re actually dealing with is criticism. Some people call it ‘critique’ to make it sound better but it’s still criticism. Criticism is not neutral and so it has lots of emotion attached.

In the last decade I joined a closed online short story writing group of around a dozen fiction writers. We all knew each other online through blogging and wanted to improve our writing. The idea was that we would each write and share a story once a fortnight. The stories were posted anonymously by one of the group – we took turns – and the others would give feedback. To begin with we only gave positive feedback until one of us pointed out that we weren’t going to get very far that way. We were a bit scared about being more critical, but gradually our feedback became more robust, with honesty about the elements of each story that didn’t work for us and why, as well as praise for the parts that did and suggestions for how to overcome weaknesses. We built up a lot of trust in that group and it helped us to give better feedback and so become better writers.

This experience taught me that trust is important to effective feedback. In the group we built trust over time. If you’re writing an anonymous peer review, you need to create trust all at once.

Another thing that is important is blending praise where possible, or at least advice, with your criticism. I had a review for the typescript of my last book which was entirely critical. Essentially, it said the book was rubbish and should never be published. The reviewer is entitled to their opinion, and I have been a writer for far too long to be upset by critical feedback, but the problem was that the review gave me no help at all. There was nothing in it which I could use to improve my writing. (Luckily I had two other reviewers at that stage who took a more balanced approach and did give me constructive criticism, advice, and some praise.)

So, from all my years of experience of receiving and giving feedback on writing in several genres, here are my twelve top tips for giving good quality feedback that others will trust.

Be honest in all the feedback you give.
Read the piece you’re giving feedback on carefully, thoroughly, at least twice.
While you read, make notes of thoughts that occur to you. As a minimum, these should include: aspects of the work you think are good; where you think there is room for improvement; anything you don’t understand; references the author might find helpful.
Be sure to praise the good points in the author’s work. This helps to build trust and also lets the author know what they can relax about.
Be open about anything you don’t understand. Doing this worries some people because they think they may look stupid, particularly if they’re giving feedback to a peer or colleague rather than writing an anonymous review. But it’s really helpful feedback for writers because it may be that they haven’t written clearly enough.
Give a straightforward assessment of areas where you think there is room for improvement.
Tell the author how you think they can improve their work. This is crucial. If you’re only saying where improvement is needed, you’re only doing half the job.
Where relevant, suggest references the author has missed.
If you think extra references would be helpful but nothing specific springs to mind, have a quick look on a website such as Google Scholar or the Directory of Open Access Journals and see if you can find something to point the author towards.
Don’t worry if you can only offer a certain amount of help because of the limits to your own knowledge. It’s fine to say, for example, that a quick online search suggests there is more relevant literature in the area of X; you’re not certain because X lies outside your own areas of interest but you think it would be worth the author taking a look.
Acknowledge the author’s emotions. For example, after giving quite critical feedback, you might say something like, “I realise that implementing my suggestions will involve a fair amount of extra work and this may seem discouraging. I hope you won’t be put off because I do think you have a solid basis here and you are evidently capable of producing an excellent piece of writing.” (Though remember #1 above and don’t say this if it’s not true.)
Be polite throughout, even if your review is anonymous. Anonymity is not an excuse for rudeness.

If there’s anything I’ve missed, please add it in the comments.

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $12 per month. If you think four of my blog posts are worth more than $12 – you can help! Ongoing support would be fantastic but you can also support for a single month if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to give financial support at this time, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

Is Peer Review Bad For Your Mental Health?

Posted by Helen Kara

peer review people I am currently waiting for peer reviews of two books I’ve worked on: one sole-authored, one co-authored. We don’t talk much about the experience of waiting for reviews, and it’s not something that appears to have been researched. Yet it’s something everyone doing academic work has to go through and it may be bad for our mental health.

I’m finding it particularly difficult at the moment because a lot rides on these reviews. The sole-authored book, which is on a contentious topic, has already had one set of reviews. Reviewer 1 was utterly damning, saying ‘I couldn’t find anything to praise’. Luckily, reviewers 2 and 3 were more measured, offering both praise and constructive criticism, and their input helped me to revise and strengthen the typescript. However, in the process, my editor and I realised that we needed further reviews from people with a particular kind of specialist knowledge. My editor approached around 10 potential reviewers, but only one agreed to do the job. So I’m gibbering – what if that person agrees with Reviewer 1?

The co-authored book is in a contentious format. My co-authors and I decided that I would be the person to liaise with publishers, as I have form in this process. The last time I liaised with academic publishers for a co-authored book was in the early 2000s, and I’d forgotten how heavily responsible it makes me feel. Fortunately, I’ve found a publisher that is interested and has sent the book out for reviews, to two professional academics and two students. This is great – and terrifying – but at least there are four reviewers. Even so, what if they ALL think it’s rubbish?

In many ways I love the peer review system. I welcome feedback on my writing, and I’m not at all averse to constructive criticism. I am by no means arrogant enough to think I can write a good book without input from others. Yet peer review, as a process, is fraught with uncertainty. Comments may not be constructive, or may not come at all. They may be positive, or negative, or in between, or a mixture.

There is a body of research which demonstrates that uncertainty has a detrimental effect on mental health. Luckily for me, my mental health is fairly robust right now, so I can use this period of uncertainty as an opportunity to build further resilience. But what about those who aren’t so fortunate? We’re hearing a lot about the mental health crisis in higher education, but nobody seems to be talking about the potential contribution of the peer review system to this crisis. Given the evidence of links between uncertainty and mental health, it seems likely that there may be a relationship here.

I’m not arguing that we should tear down the peer review system and replace it with something completely different. Time spent waiting for reviews also has a positive effect, in that it creates necessary distance between the author and their work, meaning sensible revisions are easier to make. But I do think we need to be aware of the effects of uncertainty and take steps to reduce its impact on us. Here are four ideas.

Aspects of life fall into three categories: those you can control, those you may be able to influence, and those you can neither control nor influence. Spend most of your energies on the first, some on the second, and none on the third. So I will spend my energies on hard work and good fun, and with any luck I won’t have much energy left for fruitless worry about the outcome of the peer reviews.

Plan for different outcomes. Plans for positive reviews are easy, plans for negative reviews more challenging. For me, the worst-case scenario is that the publisher decides not to publish after all, which would mean – for either book – several years of work down the pan. However, that is unlikely, and if it does happen I/we can revise and submit again elsewhere.

Acknowledge how you’re feeling. Writing this blog post is one way for me to acknowledge my own difficult feelings about this waiting period. In professional UK society the culture is not to talk about feelings much, if at all; if anyone asks how you are, the standard answers to give include ‘I’m fine’, or (with an eyeroll) ‘snowed under’. It’s as if we’re not allowed to give a real answer to the question. Yet suppressing our emotions is also bad for our mental health, so let’s talk about the difficulty of waiting, being in limbo, for unpredictable peer reviews.

Practise self-care. All the usual stuff: eat sensibly, take exercise, get enough sleep – or, if you can’t sleep, rest your body quietly in a dark room and try to still your mind. There are some good video soundtracks and podcasts online to help you sleep. Work can be part of self-care when it’s work you enjoy and you don’t do too much. Spending time with loved ones is definitely part of self-care.

The peer review system can also be hard on reviewers, such as by asking more of people who are already too busy, and offering only intangible rewards. Saying ‘yes’ to a review request adds an extra burden of work, saying ‘no’ comes with an extra burden of guilt. Some people deal with this by deciding how many reviews they will undertake, such as 12 in a year, or three per draft article or book they themselves submit. That’s a great example of focusing on what you can control.

If you’re waiting for reviews yourself, the wait will be over, sooner or later. I hope you will be able to use the advice in this post to help make the process a little easier – as I intend to do myself. I wish you luck.

The Variety Of Indie Research Work

Posted by Helen Kara

variety One of the things I love about being an independent researcher is the sheer variety of projects I work on and tasks I might do in a day. Yesterday, I was only in the office for the afternoon, yet I worked on at least seven different things. Here’s what I did.

First, I checked Twitter, and found a tweet with a link to a blog post I wrote about an event that is part of a project I’m working on with and for the forensic science community. This is a new departure for me, in that I haven’t worked with forensic scientists before, though the work itself is straightforward. I’m supporting a small group of people with research to identify the best way to create a repository for good quality student research data, and it’s surprisingly interesting. So I retweeted the tweet.

Second, I dealt with the morning’s emails. The arrival of a purchase order I’d been waiting for weeks to receive – hurrah! I formulated the invoice and sent it off to the client. Then some correspondence about the creative research methods summer school I’m facilitating at Keele in early July – just three weeks away now, so the planning is hotting up (and there are still some places left if you’d like to join us – it’ll be informative and fun). The most interesting email was a blog post from Naomi Barnes, an Australian education scholar who is considering what it means to be a white educator in the Australian school system. This chimes with the work I am doing on my next book, so I leave a comment and tweet the link.

While on Twitter, I got side-tracked by a tweet announcing #AuthorsForGrenfell, an initiative set up by authors for authors to donate items for auction to raise funds for the Red Cross London Fire Relief Fund to help survivors of the Grenfell Tower fire. I’d been wanting to help: my father is a Londoner, I have always had family in London, I lived in London myself from 1982-1997, and one member of my family is working in the tower right now to recover bodies. So it feels very close to home. But I’m not in a position to give lots of money, so I was delighted to find this option which I hope will enable me to raise more money than I could give myself. I have offered one copy of each of my books plus a Skype consultation with each one. My items aren’t yet up on the site, but I hope they will be soon because bidding is open already. If you’re one of my wealthy readers, please go over there and make a bid!

Then I spent some time researching aftercare for data. Yes, indeed there is such a thing. So far I’ve come up with two ways to take care of your data after your project is finished: secure storage and open publication. They are of course diametrically opposed, and which you choose depends on the nature of your data. Open publication is the ethical choice in most cases, enabling your data to be reused and cited, increasing your visibility as a researcher, and reducing the overall burden on potential research participants. In some cases, though, personal or commercial sensitivities will require secure storage of data. There may be other ways to take care of data after the end of a project, and I’ll be on the lookout for those as I work on my next book.

By now it was 6 pm so I did a last trawl of the emails, and found one from Sage Publishing with a link to a Dropbox folder containing 20 research methods case studies for me to review. They publish these cases online as part of their Methodspace website. I like this work: it’s flexible enough to fit around other commitments and, like other kinds of review, it tests my knowledge of research methods while also helping me to stay up to date. Best of all, unlike other kinds of review, Sage pay for my expertise. So I downloaded all the documents, checked and signed the contract, and emailed it back with a ‘thank you’. By then it was 6.30 pm and time to go home.

As the old saying goes, variety is the spice of life. I certainly like the flavour it gives to my work. Some days I work on a single project all day; those days are fun too. Yesterday I worked in my own office, today I’m out at meetings locally, tomorrow I’m off to London. It’s always ‘all change’ and I wouldn’t have it any other way.

First Peer Review – And Pizza

Posted by Helen Kara

I’ve just done my first ever proper grown-up peer review of an article which had been submitted to a research methods journal. It was an interesting intellectual and emotional experience.

The article’s title was enticing and promised new insight into an aspect of research methods. I looked forward to reading it and learning about a point I hadn’t previously considered. And the content did deliver new insight – but it delivered it really, really badly. It was such a disappointment, like getting the pizza you ordered an hour late, cold, and with your favourite ingredient missing.

I felt such empathy for the author. He or she had obviously put in a fair amount of effort, and was going to be bitterly disappointed by my review. I could feel that pain. But it wasn’t a borderline decision; the article needed a lot more work. Key references had been left out – imagine an article on psychoanalysis that doesn’t cite Freud or Jung, and you’ll get the picture. Also, the argument made was woefully under-theorised, and with very little interpretive analytic work either.

I had prepared carefully, reading the journal’s guidelines for reviewers, the COPE guidelines for reviewers, and Pat Thomson’s posts on reviewing journal articles, while reflecting on my own experience of receiving reviews. So I wrote as constructively as I could, giving praise where I thought it was merited, specific references to help the author build the context, and suggesting some questions they might consider at the interpretive stage. I was quite relieved to find I was up to the job. But I still felt bad for the poor author.

I also felt bad for the entire research profession, because the argument being made was a really important one which needed to be heard. So much so that I thought I might have difficulty keeping it confidential. It was one of those points which, once stated, seemed blindingly obvious. And we all read so much. I was determined to maintain my professional standards and protect the author’s intellectual property – yet I could imagine, a few months on, absent-mindedly saying, ‘Oh yes, I read something somewhere, can’t quite remember where, but it made this really good point…’ I wonder how often that happens.

Then I had an idea. I hopped onto the web and did a quick search – and lo and behold, an article was published just a couple of months ago, making the same point. Whoopee! I could talk about it after all! But oh no… my poor, poor author…

Actually, it’s not a disaster, because there is more than enough room for two articles making this point in different journals. So I added that reference, too, to my review, and ended with some words of encouragement. I truly think this is an important article in the making, and with a little more work, the author could deliver his or her argument in an article which is more like a warm, fragrant, appetising pizza, made from good ingredients in the right proportions.