Further reflections on the refereeing process


Not so long ago, I wrote about some of the issues associated with peer reviewing of research grant proposals.  This morning, I received editorial comments on one of my recently submitted papers – four sets of comments were broadly supportive, usefully recommending changes that would improve the paper.  However, a fifth referee clearly had not understood the purpose of the paper, which was a largely qualitative analysis of ICTs and disability in Ghana.  This is what the referee wrote:

“The paper lacks a profound research method & data analysis techniques.
In order to improve the paper I suggest:
-You develop taxonomy of the various possible factors (drivers, benefits, barriers, pitfalls) related to:
ICT & Special Education Needs in Developing Country Settings.
-Make a thorough field study grounded by previously derived taxonomy
-Use statistical analysis to determine the correlations between the taxonomies & derive the hypothesis for the study. Or use grounded theory analysis if you are interested more in the phenomenon rather than the correlations.
For the time being the paper findings are scattered and cannot be granted as validated or evenaccurate or complete.
Therefore the paper is not ready yet for publishing”.

OK – at one level, I accept that there are indeed different approaches to intellectual enquiry, but it seems quite clear that this referee fails to see the value of qualitative approaches, and is seeking to impose one particular view of the research process.

At least the other referees found something that they liked in the paper:

  • “This article addresses a particularly important issue very well. The authors understand the problem deeply and support their case with relevant evidence and clear writing.”
  • “This manuscript addresses an important and inadequately addressed topic. Data presented is valuable in informing programs and policy needs related to ICT for people with disabilities in educational settings in Ghana and other low-resource communities.”

I am tempted entirely to give up sending papers to academic journals – let’s face it, few people read them anyway – and instead simply put out material on the Internet and see what readers themselves make of them!

At the very least, I will try in future to submit papers to journals where I have greater faith in the quality of the refereeing process!

———————————————

Following correspondence with the journal’s Editor in Chief, I am delighted to say that my co-author and I are resubmitting our paper, and will include with this a commentary of exactly what we think of the referee’s comments above. Let’s see what happens!

About these ads

7 Comments

Filed under Ethics, Higher Education

7 responses to “Further reflections on the refereeing process

  1. If what you’ve just quoted is the full review you got, it definitely seems as they didn’t give a damn about your methodology and tried to impose theirs instead.

    It would nevertheless be worth taking into account the approach of the journal, that is, if it were a pro-quantitative analysis journal, then the review would make more sense.

    In any case, we do need to rework the whole system, starting with reviewing the reviewers ;)

    • Srinivas

      I have received a similar set of comments about paper submitted for ICTD research conferences.

      Broadly what reviewers are asking for in the ICTD field are

      1. Stated objectives, explicit research questions, frameworks
      3. Citation of previous ICTD studies–I find this ridiculous that the Reviewers do not seem to get that not citing someone may have to do with disagreement or that R. Heeks may not have anything relevant to say in the paper under review
      4. Emphasis on abstract and grand theory building about “telecenters” as against attention to local context.
      5. Case studies are expected to have simple narratives of either “successful” or “unsuccessful” projects.

      ICTD project cause minor and often unintended changes and their failures are for the most part attributable to unintended consequences.

      In my opinion, qualitative research is better equipped to capture them. But clearly I am in minority.

      After some initial promise, ICTD field has come to resemble the U.N. The important people in it take themselves very seriously. Life in the rest of the globe goes on.

    • unwin

      Hi Ismael

      Yes, this was the full referee’s report – that was all they wrote – and apart from anything else I hope you would agree that it is badly written!

      Any good referee would have taken more care and attention – and would have been open to differing views.

      This is academic fascism of the worst order! ;-)
      Tim

  2. Thao

    I really, really, really, really like this:

    “I am tempted entirely to give up sending papers to academic journals – let’s face it, few people read them anyway – and instead simply put out material on the Internet and see what readers themselves make of them!”

    Does this mean I can now stop working on the 2 journal papers I’m bored with?

    What you said made sense – given that the process of submitting to journal is time consuming to say the least, the return on investment of time is quite dismal. I’m not excited with the prospect of working very hard on a paper, just to have it skim-read by 6 other academics.

    Then why do academics keep working on journal papers? Because it’s the expected output of my research project and one of the criteria for measuring academic performance? Because of the opportunity of having your work examined through a rigourous peer review process? But conferences are better at this, and you get feedback online too.

    What if I don’t share the results of my research in any journal papers and instead investing my time in sharing the findings in blogs and at conferences?

  3. Bjorn Everts

    Then there is the question of access. Is it ethically sound for us to sell the rights to our ict4D findings to an academic journal that only an elite group have access to?

    As a colleague pointed out to me “You want to publish those articles if you want an academic career”.

    What is more important, career or work that gets read by people who could benefit from it?

    Blogs would be a great way to get ideas out there but they are not really adding the “the body of literature”.

    Is there a way that we can create an online platform that ensures academic credibility, but lightens the process and increases accessibility?

  4. unwin

    Hi Bjorn

    Perhaps this is something we can discuss further at the PSM tomorrow! In a nutshell, many online journals have explicitly been set up to try to make knowledge freely and more easily available along the lines you suggest.

    Trouble is, that still does not get round poor refereeing!

    With the Collective, I have tried to encourage people to make material available through our working papers page – it would be nice to see more things made available there.

    Tim

  5. Pingback: On publishing in ICT4D | Tim Unwin’s Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s