How To Check An Index

index checkingIn August 2012 I was eagerly awaiting publication of my first research methods book, Research and Evaluation for Busy Students and Practitioners: A Time-Saving Guide (now in its second edition). I received an email asking me to check and give feedback on the attached draft index. I had absolutely no clue how to check an index. It looked like a credible index to me so I sent an email back saying thanks, it looks great, and hoped that would pass muster.

In May 2014 I was delighted to receive another email telling me that the book had been positively reviewed in the International Journal of Social Research Methodology. Although the review was indeed predominantly positive, the reviewer – as reviewers will – offered some criticism too. For example, she stated that, in contradiction to the book’s title, evaluation had only been mentioned once in its pages. Almost two years had passed since I’d worked on the manuscript and I began to doubt myself so I turned to my copy to check. I was reassured to find evaluation mentioned on many pages. But then I wondered, how could the reviewer have made such an error? The rest of her review suggested that she had read the book quite carefully. I turned to the index – and found that there was only one page number given for ‘evaluation’.

I could argue here that the reviewer should have been more careful, or that the indexer should have been more thorough. But actually I think it’s my fault because I didn’t know how to check an index. On the plus side, this is a useful cautionary tale which demonstrates that indexes are used by many people in many ways. This is something that indexers understand, though they are not infallible and will never know a book as well as its author – which is why authors are asked to check indexes. But nobody ever explains how to check an index. So I’m going to try to do just that. I’m still no expert, but I have learned some points I can share.

There are three key points to consider: what the index does for the book, what the index does for the book’s readers, and whether the index is a good index by the standards of other indexes. These can be converted into three questions:

  1. Does the index accurately reflect the content of the book?
  2. Does it do so in a way that will make sense to your readers?
  3. Is the index, in itself, a good quality index?

To answer the first question, begin by making a list of key words from your title, chapter headings, and sub-headings. Ensure all of those words are properly and fully represented in the index. If they’re not, don’t try to fix it yourself or even make suggestions about how to fix the problem. Simply explain to the indexer which words need more prominence and why. Then let them sort it out because they will be able to do so far more quickly and effectively than you.

Once you’ve done that, read through the index with your book’s readers in mind. Is the language of the index closely aligned with the language of the book? Are the headings and sub-headings concise and useful? Is the index logically organised and easy to read? Are there double postings when necessary, e.g. ‘data: quantitative’ and ‘quantitative data’? Is the punctuation clear and consistent?

Then consider the more detailed indicators of index quality, usefully set out by the American Society for Indexing. For example:

  • Do main headings or sub-headings have more than 5-7 page numbers attached? If so, they may need to be broken down further.
  • Are there a reasonable number of sub-headings for each main heading? If there are more than a column’s worth then some may need to be combined.
  • Are sub-headings at a sensible level? If not, revision may be needed.
  • Are the page numbers accurate? Spot-check some to make sure.

If you want to know more, the ASI have also produced a book on the subject: Indexing for Editors and Authors: A Practical Guide to Understanding Indexes. I haven’t read it myself yet but it looks comprehensive and useful. (Thanks to Nicola King aka @icemaiden1964 for pointing me to these resources on Twitter.)

When my second edition index arrived and evaluation still didn’t have a high profile, I asked the indexer to make appropriate amendments. Which she did, quickly and cheerfully.

These days I feel more confident when I receive an index to check. I hope you will too.

If you have any good index-related stories to tell, please share them in the comments.

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $34 per month. If you think a day of my time is worth more than $34 – you can help! Ongoing support would be fantastic but you can also make a one-time donation through the PayPal button on this blog if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

Academic Publishers and Production Values

pile of booksLast week a book review I wrote was published on the LSE Review of Books blog. (This review is part of the ‘review a book a week’ series I’m running through 2019.) The book I reviewed was The Lost Ethnographies, edited by Robin James and Sara Delamont, and it is an excellent collection with only one problem: poor production values.

In publishing, “production values” is a term that covers the technical parts of the process. These include such things as: paper quality; page layout and cover design; font types and sizes; proof-reading, copy editing and indexing; print quality – essentially all the different factors that go into making a physical or digital book. A publisher with high production values is one that aims for good quality in these factors; a publisher with low production values is the opposite.

The Lost Ethnographies was published by Emerald and I’m sorry to say the production values weren’t great. As a reviewer, this was handy because it gave me something to criticise, but as a reader it was intensely irritating. There were typos on most pages, the print quality was poor, and the index was inadequate. I’m seeing more and more of this with academic books and it’s beginning to annoy me.

I understand from people who work in publishing that some academic publishing, particularly of monographs, is uneconomic. Therefore they have to outsource proof-reading, omit indexes, keep paper costs to a minimum, and so on. I hear from academics that they are really fed up with having to spend time, sometimes a lot of time, on correcting the errors of incompetent copy editors and proof-readers. At times these people are even introducing errors into books and articles. Here are some examples from the last week’s conversations:

“I had a difficult relationship with the people [publisher] outsourced editing to in [overseas country] – big issues were introduced the first time I got the proofs (bits missing, new wrong spelling) and it took a lot of pushing from me to get them changed.”

“I did an article on Jewish [redacted] whose editor changed every mention of midrash to mid-rash. It makes it sound as if I were writing about the aetiology and progression of measles.”

“When I first started writing and publishing I didn’t know how awful it was and consequently I didn’t proof read as carefully. Any newer academics who trust the process will find things are being published with typos, added words and other random deletions and insertions that ruin their papers. It is definitely getting worse and taking hours of my time to undo the damage at proofing stage.”

Worse still, academic publishers with low production values have the gall to charge three-figure sums for their books. From what I hear, Springer, Emerald, Palgrave and Routledge have bad reputations in these areas, while smaller academic publishers, such as Policy Press and Jessica Kingsley have much better production values and pricing policies.

In theory, the trade-off with the bigger publishers is that they’re better at distribution, marketing, and selling translation rights, but in practice this may not be the case. I am also hearing that even getting commitments about things like marketing and pricing into contracts with large publishers may not mean they are met. I heard one sad tale last week about pricing, where the author fought hard to have their book reasonably priced as per their contract, but didn’t have much success. I heard another about a publisher who had made clear commitments on marketing in a publishing contract but then didn’t see them through. The author concerned did what they could to put pressure on the publisher, but couldn’t afford to hire lawyers and in the end had to put up with broken promises and shattered dreams.

It seems it’s no longer the case that authors simply write books and publishers do the rest. It also seems that we have reached a point where academic monographs are being published badly because they are uneconomic. There is a simple solution to this: self-publishing. Perhaps it is time for academic researchers to build self-publishing costs into their funding bids. Authors could commission their own copy editors, proof-readers, indexers, page layout specialists and cover designers. That way they could have full control of the process and ensure that their book’s production values are high. Copy editors and proof-readers can be found via the Society for Editors and Proofreaders and indexers via the Society of Indexers. Page layout specialists and cover designers don’t yet have professional associations, so look for people with experience of academic work and testimonials that you can check such as Blot Publishing, or ask around for a recommendation.

Of course self-publishing isn’t valued by the REF, so some UK-based academic authors will have to continue working with commercial publishers. But I think that might change in time. Also, there are no paywalls for self-published books and articles. Digital self-published materials such as e-books and pdfs can be made available to readers for free, and hard copies can be produced as print-on-demand for small sums to cover costs. So there is a strong argument for self-publishing being the ethical option. (And blogging is self-publishing too!)

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $34 per month. If you think 4-5 of my blog posts is worth more than $34 – you can help! Ongoing support would be fantastic but you can also make a one-time donation through the PayPal button on this blog if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

In Praise Of Academic Re-Reading

Fields Of Play coverI read novels for pleasure, and I often re-read novels for pleasure too. I’ve read all Terry Pratchett’s books, and if I’m a bit down or feeling overwhelmed, a re-read of one of those will always cheer me up. I sometimes revert to the comfort of children’s books when I’m poorly: Susan Cooper’s The Dark Is Rising series is a great favourite. Then there are books I re-read because they’re simply too good to read only once, such as Keri Hulme’s The Bone People which I re-read every few years.

Right now, though, I’m doing something I don’t usually do: I’m re-reading an academic book. It’s Fields of Play: Constructing an Academic Life by Laurel Richardson. Richardson is an American feminist sociologist and her book came out in 1997, two years before I started my MSc in Social Research Methods. I read it first for that course, admired and loved it, and have referred to it often since then. But it never occurred to me to re-read the book until now.

I chose to re-read it because I’m embarking on a new writing project focused on creative writing in academia. I knew I wanted to draw on Richardson’s work, and I thought to myself that I should re-read her book. You know what? This is the very first time it has ever occurred to me to re-read an academic book. I have occasionally re-read an academic journal article, but I don’t do that often either. Yet I regularly re-read novels. So why is this?

I think there are a few reasons. First, novels are stories, and stories are essential. They’re important for my wellbeing in a very different way from academic literature. I could live without academic literature much more easily than I could live without stories. Second, let’s face it, some academic books aren’t particularly enjoyable or interesting to read. Third, not all academic books need reading from cover to cover in the first place. For example, some are reference books to dip into, others are edited collections where not all chapters are equally relevant to each reader.

But then there are the other books: the ones that are engaging and inspirational, exciting and even at times hard to put down. Fields of Play is one of those. It’s a fabulous book. When I first read it, it was radical, inspiring, full of feminist rage and joy which spoke to me as clearly as the concepts and arguments set out by the author. Richardson dismantles the rationale for conventional academic writing with its passive voice and authorial authority. Then she creates a rationale for using fiction techniques, poetry, drama and other creative approaches in academic writing. And she practises what she preaches within the text, to excellent effect.

Reading this book again after almost 20 years, I find there is very little that has dated. Richardson’s experiences of discrimination at the hands of male colleagues are similar to those I hear of regularly from women in academia today. I’m also aware that the fight against conventional academic writing continues, as I frequently hear from doctoral students in despair because their supervisors won’t let them write in the first person. These are disheartening messages. But they also mean that this angry, loving book is still highly relevant.

I’m really happy to be re-reading this book. I’m learning new things because of course I have a very different context for Richardson’s work than I did two decades ago. So when I’ve finished this one, I’ll be thinking about other academic books I’ve loved and might be glad to re-read. But in the meantime, I wonder if there are any academic books that you re-read, as opposed to dipping in and out for reference. Maybe everyone is a re-reader except me! If you do re-read, I’d love to know which books you return to, if you could take the time to leave a comment.

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $32 per month. If you think 4-5 of my blog posts is worth more than $32 – you can help! Ongoing support would be fantastic but you can also make a one-time donation through the PayPal button on this blog if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

Ten Top Tips For Managing Your Own Research

crossroads-1580168__340When someone mentions research methods, what do you think of? Questionnaires? Interviews? Focus groups? Ways of doing research online? Do you only think of data gathering, or do you think of methods of planning research, analysing data, presenting and disseminating findings?

Research methods is a huge and growing field with many books and innumerable journal articles offering useful information. But nobody talks about methods for managing your own research. Perhaps you’re doing postgraduate research in academia or workplace research such as an evaluation. Even if you’re a fully funded full-time doctoral student, research is not all you do. Research has to fit in with the rest of your life and all its domestic work, family needs, other paid or voluntary work, hobbies, exercise, and so on.

Nobody talks about the methods for doing this kind of personal research management. Or, at least, not many people. I said quite a lot about it in my book Research and Evaluation for Busy Students and Practitioners. Petra Boynton also addresses it in her book The Research Companion. But I haven’t seen it mentioned anywhere else (if you have, please let us know in the comments). So here are ten top tips:

  1. Plan everything. Lots of books will tell you how to plan your research project. What they don’t say is that you also need to plan for the changes to your life and work which will result from you taking on the research. How will your research affect your other commitments? What do you need to do to minimise the impact of your research on your other commitments and vice versa? Build in contingency time for unforeseen events.
  2. Manage your time carefully. Use your plan to help you. Break down the main tasks into monthly, weekly and daily to-do lists. Review these regularly.
  3. Learn to work productively in short bursts. It may seem counter-intuitive, but most people get more done this way than by setting aside whole days to work on a project.
  4. Use time when your mind is under-occupied, e.g. when you’re waiting in a queue or doing repetitive household tasks, to think about and solve problems related to your research.
  5. Seek support from your family. Make sure they know about your research and understand its importance to you.
  6. Seek support from colleagues, managers, tutors etc, whether your work is paid or unpaid. Make sure they know about your research and understand its importance in your life.
  7. Don’t cut corners in ways that could damage your health. Eat sensibly, take exercise, get enough sleep and rest.
  8. Take breaks. At least three short breaks in each day, one day off in each week, and four weeks off in each year.
  9. Don’t beat yourself up if things go wrong. Be kind to yourself and learn what you can from the experience. Then re-group, re-plan, and set off again.
  10. Reward yourself appropriately for milestones reached and successes achieved.

In my view, these are as much research methods as questionnaires and interviews. Learning to use them involves acquiring tacit knowledge. I’ve been on a mission to convert tacit knowledge to explicit knowledge ever since I started writing for professionals. This blog post is part of that process. If you have other tips, please add them in the comments.

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $12 per month. If you think 4-5 of my blog posts is worth more than $12 – you can help! Ongoing support would be fantastic but you can also support for a single month if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

How To Give Feedback On Academic Writing – Twelve Top Tips

feedback peopleA recent discussion on Facebook reminded me that I’ve written about how to deal with feedback from reviewers, but I haven’t written about how to give feedback to peers and colleagues. There is an art to this which I have learned, paradoxically, from receiving feedback, which taught me what helps and what does not help.

Feedback is a fairly neutral word but what we’re actually dealing with is criticism. Some people call it ‘critique’ to make it sound better but it’s still criticism. Criticism is not neutral and so it has lots of emotion attached.

In the last decade I joined a closed online short story writing group of around a dozen fiction writers. We all knew each other online through blogging and wanted to improve our writing. The idea was that we would each write and share a story once a fortnight. The stories were posted anonymously by one of the group – we took turns – and the others would give feedback. To begin with we only gave positive feedback until one of us pointed out that we weren’t going to get very far that way. We were a bit scared about being more critical, but gradually our feedback became more robust, with honesty about the elements of each story that didn’t work for us and why, as well as praise for the parts that did and suggestions for how to overcome weaknesses. We built up a lot of trust in that group and it helped us to give better feedback and so become better writers.

This experience taught me that trust is important to effective feedback. In the group we built trust over time. If you’re writing an anonymous peer review, you need to create trust all at once.

Another thing that is important is blending praise where possible, or at least advice, with your criticism. I had a review for the typescript of my last book which was entirely critical. Essentially, it said the book was rubbish and should never be published. The reviewer is entitled to their opinion, and I have been a writer for far too long to be upset by critical feedback, but the problem was that the review gave me no help at all. There was nothing in it which I could use to improve my writing. (Luckily I had two other reviewers at that stage who took a more balanced approach and did give me constructive criticism, advice, and some praise.)

So, from all my years of experience of receiving and giving feedback on writing in several genres, here are my twelve top tips for giving good quality feedback that others will trust.

  1. Be honest in all the feedback you give.
  2. Read the piece you’re giving feedback on carefully, thoroughly, at least twice.
  3. While you read, make notes of thoughts that occur to you. As a minimum, these should include: aspects of the work you think are good; where you think there is room for improvement; anything you don’t understand; references the author might find helpful.
  4. Be sure to praise the good points in the author’s work. This helps to build trust and also lets the author know what they can relax about.
  5. Be open about anything you don’t understand. Doing this worries some people because they think they may look stupid, particularly if they’re giving feedback to a peer or colleague rather than writing an anonymous review. But it’s really helpful feedback for writers because it may be that they haven’t written clearly enough.
  6. Give a straightforward assessment of areas where you think there is room for improvement.
  7. Tell the author how you think they can improve their work. This is crucial. If you’re only saying where improvement is needed, you’re only doing half the job.
  8. Where relevant, suggest references the author has missed.
  9. If you think extra references would be helpful but nothing specific springs to mind, have a quick look on a website such as Google Scholar or the Directory of Open Access Journals and see if you can find something to point the author towards.
  10. Don’t worry if you can only offer a certain amount of help because of the limits to your own knowledge. It’s fine to say, for example, that a quick online search suggests there is more relevant literature in the area of X; you’re not certain because X lies outside your own areas of interest but you think it would be worth the author taking a look.
  11. Acknowledge the author’s emotions. For example, after giving quite critical feedback, you might say something like, “I realise that implementing my suggestions will involve a fair amount of extra work and this may seem discouraging. I hope you won’t be put off because I do think you have a solid basis here and you are evidently capable of producing an excellent piece of writing.” (Though remember #1 above and don’t say this if it’s not true.)
  12. Be polite throughout, even if your review is anonymous. Anonymity is not an excuse for rudeness.

If there’s anything I’ve missed, please add it in the comments.

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $12 per month. If you think four of my blog posts are worth more than $12 – you can help! Ongoing support would be fantastic but you can also support for a single month if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to give financial support at this time, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

Independent Research, Writing, and Financial Reality

money twenty pound notesEvery so often I post about how much money I make. As I’m just finishing my 2017-18 accounts, it seems a good time to update this.

I have written before about the difficulties the recession caused to my business and the bumpy road back to reasonable prosperity. In 2017-18 I invoiced for £34,338.54 of business, a bit down on the 2016-17 figure of £39,939 though that was partly because I took on a sizeable contract in the spring of 2018 but didn’t receive my first payment instalment until after my year end on 31.7.18.

The amount I invoice for is representative of the amount of work I do, not the amount of money I have in my pockets. In 2016-17 my post-tax profit was £14,057 – and I was able to pay myself a bit more than that because I’d had an even better year in 2015-16, as reported in my earlier post. In fact, 2015-16 was by far the best year of the last 8 years.

So it’s still bumpy, but the bumps are evening out, and I’m beginning to feel that I’m back on my financial feet (except when I think about my pension plans, eek, must do something about that). It helps that my mortgage is paid off, I’m happily child-free, and I don’t have expensive tastes. Also, I have plenty of work scheduled in for early 2018. For the first time in eight years, I don’t feel as if I should spend every spare moment trying to generate work.

Also, my research business doesn’t represent the whole of my income. There is also the income I derive from writing, which in 2017-18 was royalties of £1,663.70 from my trade published books and £306.25 from my self-published books, plus £268.64 from the wonderful ALCS. That’s a total of £2,238.59 for the year – though again there were outgoings to set against that: memberships of the Society of Authors and the Textbook and Academic Authors’ Association, royalties to Nathan Ryder who co-authored Self-Publishing for Academics, and all the books I bought. Altogether that comes to £593.48 and brings down my writing-related income to £1,645.11. Which is enough to pay for a month of writing time. I have to look at it that way, and not think in terms of an hourly rate, or I’d never write another word… if I wasn’t a writing addict.

Writing income is bumpy too. As my trade royalties arrive annually in October, I already know that they are lower in 2018-19 (£947.46) and I don’t really understand why. But I have a new book out this month, and I’ll have two short books out next month in the new series I’m working on for SAGE, plus two more next July, and I’m also co-editing and writing for a new series for Routledge, and have three other book proposals in the pipeline. The SAGE and Routledge books come with small advances totalling £1,250 so far, so in this financial year I’ve already made more from those than from the royalties on my published books. I’m hopeful that perhaps by 2021 I’ll make enough to buy myself out for two months of writing time. At that rate it should only take another 30 years of work to be able to write full-time, so it doesn’t look as though I’ll achieve that dream, as I’ll be 87 in 2051!

Sometimes people think that because my day rates are comparatively high, I must be rich. In fact, my day rates don’t only cover a day’s work, they also cover holidays, sickness and bereavement leave, time spent on unpaid but essential work such as admin and accounts, travelling time, business expenses such as heat and light and IT equipment and accountants’ fees and so on, and of course tax to be paid.

There are independent researchers who make more money than me – I know of one who is registered for VAT, which suggests they turn over more than £85,000 per year, but they work very hard for that, travelling all around the world for most of the year. That may sound delightful and glamorous but I can assure you that travelling for work, while it does have lovely moments, is mostly about trains, planes, taxis, hotel rooms and classrooms or meeting rooms. I like to work overseas, and could probably make more money if I did more of it, but once or twice a year is about right for me.

I think it is important to be open about how much money I make overall, not least because so many people ask me what it’s like to be an independent researcher. For me, it’s a terrific lifestyle, but it wouldn’t suit everyone. I’d say it’s probably as difficult as being an academic or practice-based researcher but the difficulties are in different places. If it’s an option you’re considering, you need to be as realistic as possible about the financial side.

This blog is funded by my beloved patrons. It takes me around one working day per month to post here each week. At the time of writing I’m receiving funding of $11 per month. If you think four or five of my blog posts are worth more than $11 in total – you can help! Ongoing support would be fantastic but you can also support for a single month if that works better for you. Support from Patrons also enables me to keep this blog ad-free. If you are not able to support me financially, please consider reviewing any of my books you have read – even a single-line review on Amazon or Goodreads is a huge help – or sharing a link to my work on social media. Thank you!

The Ethics of Independent Research Work #1

ethicsI guess we all know by now that I bang on a fair bit about research ethics, but I haven’t written about the ethical aspects of working as an independent researcher. I have come up with ten ethical principles for indie researchers. Many of these no doubt apply to other forms of self-employment too, but they definitely all apply to independent research work. This post contains the first five principles; I will post the other five next week.

  1. Be honest about what you don’t know

If a client says, ‘You know the legislation that…’ and you don’t, it’s best to say so. It can be tempting to nod while making a mental note to look it up online later, but that can lead to disaster. People often fear that saying they don’t know something will make them look stupid, but paradoxically the reverse is true. If you are clear about what you do know and honest about what you don’t, you will build trust with your clients much more quickly and effectively.

  1. Be clear about your capacity

Allied to this: don’t take on work you haven’t got time to do, because that won’t do anyone any favours. You won’t produce your best work for your clients, and you’ll end up burned out. OK there are times where you may choose to work at maximum capacity for a short time, e.g. as one contract ends while another begins, or to fit in a quick piece of work for a valued client. But keep these brief and infrequent, and make sure you build in recovery time. Independent research is a great career (at least, in my view), but no career is worth damage to your health and relationships.

  1. Charge a fair rate for the job

If possible, find out what the going rate is, and charge that. The going rate will vary across sectors and between countries. I have written before about how I charge for work: in brief, I charge less for charities and longer projects, more for universities, governments, and work I don’t really want to do.

Also, don’t take on jobs with inadequate budgets, unless you’re desperate for the money and prepared to accept a very low day rate. I’ve been offered a three-year national evaluation with a total budget of £5,000. Perhaps someone ended up doing that work for that money, but they would either have done a very poor job or effectively accepted an extremely low day rate.

  1. Don’t accept work on an unethical basis

One potential client rang me towards the end of the financial year to ask if I could invoice her for several thousand pounds that she had left in her budget. She said she was a bit busy, so could we sort out what I would do for the money at a later date? I didn’t know her so I asked why she had rung me. She told me she had wanted person A, but they were too busy so they suggested person B, who couldn’t take it on either and suggested me. Nowadays I would probably say a simple ‘no’, but it was early in my career, and person B was quite influential. I agreed to invoice, but only after meeting with my potential client to decide whether we could work together and what I would do for her.

Another time a commissioner rang me to ask me to evaluate a service because he wanted to close it down. I said I would evaluate the service if he wished, but I would not pre-determine the findings; they would be based on my analysis of the data I gathered. He agreed to this. I did the evaluation, and found – unequivocally – that the service was highly valued and doing necessary work. The commissioner paid my invoice, then found someone else to do another evaluation saying the service should be closed down, whereupon he closed it down. Again, with the benefit of hindsight I probably should have said ‘no’ to the assignment, but I naïvely thought that if I did the research the commissioner would abide by the findings.

  1. Don’t take work outside your areas of expertise

You may have more than one area of expertise. I have a few: children/young people/families, housing/homelessness, substance misuse, volunteering, service user involvement, third sector, training. Each of these areas formed part of my professional work before I became an independent researcher.

Earlier this decade I got an email asking me to do some work around learning disability. I replied, explaining that it was not one of my areas of expertise, and saying I didn’t think I was the best person for the job. The potential client came back saying they thought I was right and apologising for having bothered me. (I didn’t mind. I never mind answering queries about possible paid work.)

Oddly enough, a few weeks later I got another email, from someone completely different, asking me to do some work around learning disability. After rolling my eyes and thinking about buses, I sent a similar reply. This time the potential client came back saying that I sounded perfect for the piece of work they wanted to commission. They thought someone with a good knowledge of research methods but little knowledge of learning disability would bring a usefully fresh perspective to the problems they were trying to solve. Which is further evidence for (1) above.

So there you have the first five principles of ethical research work, according to me. Come back next week for the other five.

How Do Research Methods Affect Results?

questionsLast week, for reasons best known to one of my clients, I was reading a bunch of systematic reviews and meta-analyses. A systematic review is a way of assessing a whole lot of research at once. A researcher picks a topic, say the effectiveness of befriending services in reducing the isolation of housebound people, then searches all the databases they can for relevant research. That usually yields tens of thousands of results, which of course is far more than anyone can read, so the researcher has to devise inclusion and/or exclusion criteria. Some of these may be about the quality of the research. Does it have a good enough sample size? Is the methodology robust? And some may be about the topic. Would the researcher include research into befriending services for people who have learning disabilities but are not housebound? Would they include research into befriending services for people in prison?

These decisions are not always easy to make. Researcher discretion is variable and fallible, and this means that systematic reviews themselves can vary in quality. One thing they almost all have in common, though, is a despairing paragraph about the tremendous variability of the research they have assessed and a plea to other researchers to work more carefully and consistently.

One of the systematic reviews I read last week reported an earlier meta-analysis on the same topic. A meta-analysis is similar to a systematic review but uses statistical techniques to assess the combined numerical results of the studies, and may even re-analyse data if available. The report of the meta-analysis I read, in the systematic review, contained a sentence which jumped out at me: ‘…differences in study design explained much of the heterogeneity [in findings], with studies using randomised designs showing weaker results.’

Randomised designs are at the top of the hierarchy of evidence. The theory behind the hierarchy of evidence is that the methods at the top are free from bias. I don’t subscribe to this theory. I think all research methods are subject to bias, and different methods are subject to different biases. For example, take the randomised controlled trial or RCT. This is an experimental design where participants are randomly assigned to the treatment or intervention group (i.e. they receive some kind of service) or to the control group (i.e. they don’t). This design assumes that random allocation alone can iron out all the differences between people. It also assumes that the treatment/intervention/service is the only factor that changes in people’s lives. Clearly, each of those may not in fact be the case.

Now don’t get me wrong, I’m not anti-RCTs. After all, every research method is based on assumptions, and in the right context an RCT is a great tool. But I am against bias in favour of any particular method per se. And the sentence in the systematic review stood out for me because I know the current UK Government is heavily biased towards randomised designs. It got me wondering, do randomised designs always show weaker results? If so, is that because the method is more robust – or less? And does the UK Government, which is anti-public spending, prefer randomised designs because they show weaker results, and therefore are less likely to lead to conclusions that investment is needed?

And that got me thinking we really don’t know enough about how research methods influence research results. I went looking for work on this and found none, just the occasional assertion that methods do affect results. Which seems like common sense… but how do they? Does the systematic review I read hold a clue, or is it a red herring? The authors didn’t say any more on the subject.

We can’t always do an RCT, even when the context means it would be useful, because (for example) in some circumstances it would be unethical to withhold provision of a treatment/intervention/service. So what about other methods? Do we understand the implications of asking a survey question that a participant has never thought about and doesn’t care about – or cares about a great deal? I know that taking part in an interview or focus group can lead people to think and feel in ways they would not otherwise have done. What impact does that have on our research? Can we trust participants to tell us the truth, or at least something useful?

This is troubling me and I have more questions than answers. I fear I may be up an epistemological creek without an ontological paddle. But I think that bias in favour of – or against – a particular research method, without good evidence of its benefits and disadvantages, is poor research practice. And it’s not only the positivists who are subject to this. Advocates of participatory research are every bit as biased, albeit in the opposite direction. The way some participatory researchers write, you’d think their research caused bluebirds to sing and rainbows to gleam and all to be well in the world.

It seems to me that we all need to be more discerning about method. And that’s not easy when there are so many available, and a plethora of arguments about what works in which circumstances. So I think we may need to go meta here and do some research on the research. But ‘further research needed’ is a very researcher-y way of thinking, and I’m a researcher, so… does my bias look big in this?

Book Launch! And Other Events

Research ethics in the real world [FC]I am delighted to have been invited to launch my forthcoming book, Research Ethics in the Real World: Euro-Western and Indigenous Perspectives, at a seminar at City University in London on Thursday 8 Nov. This is part of a seminar series run by NatCen, City University, and the European Social Survey. I’ll be talking about why it is crucial to view research ethics in the context of its links with individual, social, professional, institutional and political ethics. I will explain why I think the Indigenous research paradigm is as important for our world as the Euro-Western research paradigm. I will outline why applying research ethics at all stages of the research process is equally essential for quantitative, qualitative, and mixed-methods researchers.

This was a much more difficult book to write than my book on creative research methods. Since that book came out, I have been asked to do a lot of speaking and teaching on creative methods. For example, I’m running an open course on creative methods in evaluation research for the UK and Ireland Social Research Association in Sheffield on 16 October, and a more academically-oriented version on using creative methods for the ESRC‘s National Centre for Research Methods in Southampton on 21 November. (And one for social work researchers in Birmingham next week, but that’s been fully booked for some time and has a long waiting list.)

If my ethics book has the same effect, I’m not quite sure how I’ll manage the workload. Still, that would be a great problem to have. In the meantime: fancy a free seminar on research ethics? Of course you do! It’s at 5.45 for 6 pm with a wine reception afterwards. I’d love to see some of my blog followers there – if you can make it, please come and introduce yourself.

Aftercare in Social Research

aftercareWhen does a research project end? When a report has been written? When a budget has been spent? When the last discussion of a project has taken place? It’s not clear, is it?

Neither is it clear when a researcher’s responsibility ends. This is rarely spoken of in the context of social research, which is an unfortunate omission. A few Euro-Western researchers recognise the need for aftercare, but they are a tiny minority of individuals. There seems to be no collective or institutional support for aftercare. In the Indigenous paradigm, by contrast, aftercare is part of people’s existing commitment to community-based life and work. Euro-Western researchers could learn much from Indigenous researchers about aftercare: for participants, data, findings, and researchers ourselves.

The standard Euro-Western aftercare for participants is to tell them they can withdraw their data if they wish. However, it is rare for researchers to explain the limits to this, which can cause problems as it did for Roland Bannister from Charles Sturt University in Wagga Wagga, Australia. Bannister did research with an Australian army band, Kapooka, which could not be anonymised as it was unique. Band members consented to take part in Bannister’s research. He offered participants the opportunity to comment on drafts of his academic publications, but they weren’t interested. Yet when one of these was published in the Australian Defence Force Journal, which was read by band members, their peers, and superiors, participants became unhappy with how they were represented. Bannister had to undertake some fairly onerous aftercare in responding to their telephone calls and letters. Of course it was far too late for participants to withdraw their data, as this would have meant retracting several publications, which is in any case limited in its effectiveness. However, particularly in these days of ‘long tail’ online publications, we need to be aware that participants may want to review research outputs years, even decades, after the substantive work on the project is done. We have a responsibility to respond as ethically as we can although, as yet, there are no guidelines to follow.

Data also needs aftercare, particularly now that we’re beginning to understand the value of reusing data. Reuse increases the worth of participants’ contributions, and helps to reduce ‘research fatigue’. However, for data to be reusable, it needs to be adequately stored and easy to find. Data can be uploaded to a website, but it also needs to be carefully preserved to withstand technological changes. Also, it needs a ‘global persistent identifier’ such as a DOI (digital object identifier) or Handle. These can be obtained on application to organisations such as DataCite (DOIs) or The Dataverse Project (DOIs and Handles). As well as enabling reuse, a global persistent identifier also means you can put links to your data in other outputs, such as research reports, so that readers can see your data for themselves if they wish. This too is an ethical approach, being based in openness and transparency.

Then there are the findings we draw from our data. Aftercare here involves doing all we can to ensure that our findings are shared and used. Of course this may be beyond our power at times, such as when working for governments who require complete control of research they commission. In other contexts, it is unlikely that researchers can have much say in how our findings are used. But we should do all we can to ensure that they are used, whether to support future research or to inform practice or policy.

Researchers too need aftercare. In theory the aftermath of a research project is a warm and fuzzy place containing a pay cheque, favourably reviewed publications, and an enhanced CV. While this is no doubt some people’s experience, at the opposite end of the spectrum there are a number of documented cases of researchers developing post-traumatic stress disorder as a result of their research work. In between these two extremes, researchers may experience a wide range of minor or major difficulties that can leave them needing aftercare beyond the lifetime of the project. For that, at present, there is no provision.

Not much has yet been written on aftercare in research. If it interests you, there is a chapter on aftercare in my book on research ethics. I expect aftercare to be taken increasingly seriously by researchers and funders over the coming years.

An earlier version of this article was originally published in ‘Research Matters’, the quarterly newsletter for members of the UK and Ireland Social Research Association.