Scientific journals constitute the main publication channel in many domains, and the international scientific community has through peer review developed a quality control system for the content of these journals. Although not always water-tight, the fact that their articles have been screened and approved by (in general) two experts in the field constitutes an external positive evaluation of the described research results. The same can be said about papers in conference proceedings, either in book or in journal format, although it is known that the peer review here is in general somewhat less strict. In all other cases, evaluators should be aware of the necessity to try to obtain themselves some expert evaluation of the quality of the paper; we give some hints for doing this below.
The international scientific community is also paying much attention to quantitative data that are supposed to measure the impact of a paper or of the journal in which the paper is published, based primarily on the number of citations that the paper or the journal receives in other publications worldwide. We refer to the page about bibliometric methods for a more detailed description and critical analysis of these methods.
We will now describe some elements that one should consider in the evaluation of a scientific publication. The reader should have no difficulty in adapting the meaning of the statements to other formats of publication, wherever the term “paper” or “article” is used.
A. The content
The true evaluation of the content of a paper can, of course, be performed only by an expert in the field. We list the most important aspects to be considered during an independent evaluation of the content:
- What is the scientific quality of the paper? Does the author have a clear understanding of the present state of knowledge in his domain? Did the author follow the appropriate steps to attain his results? Is his conclusion satisfactory?
- Was the approach of the research sufficiently comprehensive? Did the author take into account all important aspects of the problem?
- What is the originality contributed by this paper? Does the paper provide new insights in the domain it is dealing with
- What kind of impact may be expected for both the short and the long term?
B. The manuscript
Having performed successfully a high-quality research project is a necessary but not a sufficient condition for achieving a good paper. Through his publication, the researcher is sharing his findings with the worldwide scientific community, and for this reason the message should be clear and focused. Writing a good paper is an art in itself. Special points of attention are:
- Is the title well chosen (explaining the content, yet not too long)?
- Is the abstract well written and properly structured (generally: objective(s), methods, results, conclusions) and does it summarize the most important aspects of the paper?
- Are the keywords appropriate?
- Is the overall structure of the paper clear and appropriate (generally: introduction, methods, results, discussion, conclusion)?
- Does the introduction clearly identify the underlying problem in the context of the present knowledge in this domain? Does it clearly state the aim(s) of the paper?
- Does the author acknowledge related published research by others?
- Does the author clearly explain the methodology followed for tackling this problem, and the reasons for using this specific methodology?
- If the paper is based on a statistical analysis, is this performed in the correct way? Is the sampling unbiased and sufficiently large?
- How good is the quality of the figures? Is the scale of the figures clearly indicated?
- Where available, are numerical data presented in figures or tables, and are these well-organized and structured?
- Is there a database with original data linked to this paper? What is its value?
- Is the conclusion well motivated and sufficiently discussed?
- Is the language used clear and correct (vocabulary, orthography, grammar, ...)?
- Is there no unnecessary repetition of data (text, figures, tables)?
- Are the references relevant and complete, up-to-date and correctly formatted? Are the cited items publicly accessible (although not necessarily for free)?
C. Availability and visibility
- Is the selected journal the appropriate publication channel for reaching the community or public for which it is intended? (Remark: The publication in local journals – if necessary in a local language and style – may be desirable for typical development research on applied topics; this can be done in parallel with a more international publication.
- Is the journal covered by major international bibliographic databases, so that readers can easily be alerted about its contents?
- Is the journal published by one of the large international publishers or by a local enterprise (with limited distribution channels)?
- Is the paper published in an ‘open access’ journal, with global availability without delay or financial barriers? Such open access papers are available free of charge from the Internet - either via open access journals (the ‘golden road’) or via freely accessible institutional or subject repositories indexed by the major open access harvesters (the ‘green road’). While many researchers worldwide still use the selective ISI group of journals as both their main source of scientific information and as their favored outlet for publishing research results, there is an increasing amount of evidence that open access publications are generally cited more often than papers available via a paid subscription only, thus delivering enhanced visibility, use and scientific impact.
- Is the paper - simultaneously with its publication in a journal - also deposited in an institutional repository that follows the OAIMHP (Open Archives Initiative Metadata Harvesting Protocol), so that it will turn up in searches by the major international open access repository harvesters (e.g. BASE, openAIRE, OAIster) and Google Scholar?
- Does the author specify under which copyright provisions his results are published?(If nothing is specified, one should in principle ask the author’s permission before applying or further elaborating on his results, which may be an impediment to useful applications in the developing world. Many of the Open Access journals and repositories automatically include a license that allows using the results – under the condition of proper citation of the original work – as specified by one of the Creative Commons licenses.)
- Has the paper received a persistent label or identifier such as the Digital Object Identifier (DOI)?
Building the specific guide
The steps to be followed by the authors of a specific guide for evaluating scientific papers are very similar to those taken for the evaluation of other aspects: see the page on writing the specific guide.
References
|
