Things to be examined and agreed upon prior to start writing a specific guide
General
This preliminary and necessary phase concerns equally the evaluation of research, of researchers, and of scientific publications. It is dedicated to :
- think about what is expected from the evaluation : Who will be using the specific guide ? Who will be in charge ?
- make sure the conditions for a good evaluation are satisfied including – importantly - the ethical requisites,
- decide how the evaluation will be conducted.
The output of this preliminary phase will be a set of written documents, each fitting adequately the specific situation and needs, such as :
- evaluation instruments (list of criteria, questionnaires, grids, etc.)
- instructions on how to use the grid or the questionnaire,
- more generally a set of procedures and rules that the evaluators should follow.
The present page provides insights on such points. The steps to be followed when actually writing the instruments and procedure in final form are described in evaluation of research projects, the evaluation of researchers, and of the evaluation of scientific publications.
When critically reviewing or amending an existing evaluation procedure, the same phase may be applied.
Steps
In this preliminary phase it might be useful to flexibly take the following sequence:
Step 1. Define the objectives of evaluation
The evaluation objectives need to be clearly defined early in the process of writing a guide. Basically the purpose of evaluation is to help making decisions about :
- a research : assess a proposal for a new research or the extension of an on-going research, the publication of results, the application of such results, etc.
- a researcher : providing funds, awarding a degree, giving a prize, appointing or promoting the researcher, etc.
- a research team (assessing its performance mainly)
- a scientific publication
- also, in some cases, an institution (ranking of universities or departments).
Obviously the methodology for evaluating development research will depend on the purpose of evaluation. Each of these aims asks for a different and specific approach. Let us remember here that another possible objective ( a very general one, anyhow) is to contribute to maintain and increase the quality of development research. The authors of the specific evaluation guide will therefore insist that the evaluators (the users of the specific guide) would examine the questions below, and explicitly decide upon the answers. The questions are: what is it that the evaluator wants to know in this specific case ? What will be done with the answers ? In other words one needs to :
- identify what will be done with the expected outputs and outcomes of the evaluation process,
- specify whether what will be evaluated is ex ante (a research proposal) or ex post (a research already completed),
- specify whether the evaluation will apply to a given research, to a researcher or a research team,
- specify whether the evaluation is participatory or external,
- make comments, if appropriate, on possible special situations.
In most cases, obviously, a single evaluation will meet more than one objective. Examples of possible objectives are given in objectives of evaluation of development research
Step 2. Which is (or are) the disciplines(s) or categories of disciplines involved ?
It must be clear within which discipline or category of disciplines the tools and procedures will be used, and one most take into consideration the specificity of such discipline(s). A convenient beginning may be to use the three broad areas of the Academy’s operations:
- social and human sciences,
- natural and medical sciences,
- engineering and exact sciences
Other categories are listed in disciplines or groups of disciplines involved
- In each situation excess breakdown should be avoided unless there is a specific reason, which in any case needs to be justified.
- There are differences between the human sciences on the one hand vs. the natural and exact sciences on the other hand, particularly in publishing habits. Such differences will be considered when appropriate. Yet what could be very different is the set of criteria to be used, and even more the scoring system.
- The field of development research is often multidisciplinary (or better trans-disciplinary ?). In such a case the evaluators should obviously be able to use a single set of criteria reflecting the needs of the comprehensive criteria and the specificity of each discipline concerned. A negotiation may be necessary.
Step 3. Who needs evaluation ? Actors and stakeholders
The identification of actors and stakeholders is the third logical step in the building of an evaluation guide or of evaluation tools. The future users of the specific guide, in particular, need to be precisely identified.
They are people or organizations who have a stake, an interest in good development research and/or need to make decisions about research – or researchers - in this area. Their diversity is remarkable and their needs and situations also vary widely. The present chapter should therefore be interpreted and used in a flexible manner.
If necessary the responsibilities of each actor will be defined, together with appropriate terms of reference.
Step 4. Nature and type of evaluation
The nature and type of evaluation described in the specific guide will of course depend on decisions made on earlier points. The guide should in particular specify whether the evaluation will be :
- external, participatory, or both
- national or international.
In a few cases it may be appropriate to use instruments such as the Logical Framework or CPPE (Comprehensive and Participatory Planning and Evaluation) (both used successfully in a somewhat different context : the evaluation of development projects and programmes). That issue , if necessary, may be touched at this point.
In this latter case the authors of the specific evaluation guide may use the references provided in dimensions of evaluation; criteria, and decide whether such an approach is appropriate with their needs.
Step 5. Dimensions of evaluation
One of the main and earliest operation in evaluation is to choose criteria. But before making such a choice the specific guide’s author needs to determine which dimensions he/she wishes to assess. A distinction needs indeed to be made between the dimensions of evaluation and the criteria used to express such dimensions.
The Webster Dictionary provides a long list of definitions of the term “dimension”. According to the specific situation one of the following definitions may more conveniently be applied :
- “ One of the elements or factors making up a complete entity”. or
- “ The range over which or the degree to which something extends”.
A dimension can thus be expressed using one or more criteria. Good examples of how a variety of criteria can be used to express a single dimension are scientific quality or valorisation.
In the case of evaluation of development research, besides the two examples just given, dimensions such as relevance for development, performance, and impact are widely used. More dimensions are listed in dimensions of evaluation; criteria.
Step 6. Distribute responsibilities among the specific guide’s authors
A guide will seldom be built by an individual, and it more generally will be the work of a team. Various decisions will be made about the sharing of responsibilities and tasks among the authors of the evaluation guide : committee work, individual contributions, call to external referees, etc. Also, at this stage, the bases should be established for reaching a consensus. The stage is then set for starting to write the first version of the specific guide.
