Why a specific guide, and what for?
Definition
The term “specific guide” is widely used throughout the Guidelines, since the immediate objective of the Guidelines and their main output are precisely the production of “good” guides for evaluating development research. It is called a specific guide because it is intended to be used in well-defined situations, contexts, availability of resources, etc.
Some people prefer to call it a “local guide” or a “practical guide”. This is all-right, as long as the Guidelines are used to write them.
A specific guide aims to assist, in a practical manner, the person(s) in charge of evaluating a research project, a researcher or a team of researchers, or a scientific publication. Its length and content may vary widely: it doesn’t at all need to be a heavy or complicated document.
The size and the degree of sophistication of the specific guide will depend on a number of factors such as:
- its intended use: (what for? by whom?),
- the complexity of the object of evaluation,
- the requirements expressed by the evaluation’s sponsor,
- the amount of resources involved in the research and/or the resources available for evaluating it.
A specific guide can therefore be:
- just a simple questionnaire or grid, accompanied with brief instructions for its use,
- a short text with an introduction, tools and procedures, and maybe a few annexes,
- a full-fledged manual or handbook.
Their authors will decide according to their objectives and needs, but they should prefer simplicity whenever possible.
Uses
In practice the specific guide should:
- tell its reader – i.e. the actual evaluator – as precisely as possible which information he/she is expected to provide, and which tools he/she should use to that effect,
- supply the evaluator with the necessary tools and instructions,
- instruct him/her about how to design and conduct the evaluation,
- provide him/her with examples, as fit.
After having read the introduction, followed the recommendations made in the preliminary stages, and picked from other pages in the Guidelines whatever elements that are appropriate for their needs and situation, the authors of a specific guide have a clear representation of the kind, size and content of the document they are going to draft, i.e. their own guide. In the present Guidelines the case of a complete, detailed manual was chosen as a basis for guiding the authors in writing their own guide. The process will be similar – but much lighter – when designing a simpler document.
Suggested steps to be followed by the author(s) of a specific evaluation guide:
- Step 1 Preliminary stages fulfilled?
- Step 2 Questions of evaluation, dimensions, criteria
- Step 3 Select procedures
- Step 4 Conditions of evaluation
- Step 5 Actual drafting of the specific guide
Step 1. Check whether the preliminary stages have been fulfilled
The fulfilment of the preliminary stages and the decisions made at this moment are essential for the drafting of the specific guide. Prior to start constructing the specific guide, four main aspects need to be covered:
- Definition of the evaluation objectives
- Identification of the discipline(s) concerned (in the case of a multidisciplinary research, specify how to operate)
- Listing of the major stakeholders
- Type of evaluation. Define clearly whether the evaluation will be participatory, or external, or both.
Often the four points were already decided by the person or organisation which sponsors/ordered the writing of the specific guide. But even in such a case the specific guide’s author should check the preliminary stages for consistency, and make the necessary decisions.
Step 2. Formulate the questions of evaluation, decide which dimensions will be considered in the specific guide, and select criteria.
Such three essential processes are so closely linked, that they should be implemented in one, broad, iterative step.
For formulating the evaluation questions, express concisely and with simple words :
- What will the evaluator want to know about the research or the researcher?
- What for?
Then move to consider the dimensions to be used. It is becoming increasingly clear that one has to switch to a multidimensional evaluation system. Hence the need to carefully choose the dimensions to be taken into consideration in each specific evaluation guide.
Then select the criteria to be used in the specific evaluation guide, and explain:
- the choice of criteria on the basis of the selected dimensions
- the principles and rules for giving relative weights to the criteria, when necessary
Nota bene : Terms of reference
A reader of these Guidelines did mention the « terms of reference » (ToRs) as being part of the evaluation process. Actually, in our view, ToRs do not pertain to the present Guidelines : they belong to another conceptual category.
Step 3. Select and define evaluation procedures
This step will depend upon decisions made earlier, during Steps 1 and Step 2 above.
Step 4. Check whether the conditions for a meaningful evaluation are satisfied, including ethical considerations
Are the tools likely to provide meaningful answers to the questions raised about the research or researcher who is being evaluated ? Which conditions should be satisfied? This step must be included in each specific evaluation guide.
The Guidelines' users will find the necessary guidance in the Annexes under Conditions for a good evaluation. The tentative list provided in that page can be used for both designing or evaluating new instruments, as well as serving as a check-list before publishing tools and procedures. Each author should add to, or adapt the manner in which he/she will formulate the precise conditions he/she wishes to be fulfilled in each particular evaluation situation.
Basically two categories of conditions can be distinguished :
Among the operational conditions a particular aspect deserves to be stressed: consistency between the steps the user of the specific guide should follow. This can best be insured by analysing in an iterative manner whether each step followed so far (either the steps proposed above, or any other sequence preferred by the guide’s authors) is consistent with all the others. This recommendation may seem rather obvious, yet quite a few evaluation instruments generate doubts in the mind of the evaluators, precisely because of the presence of contradictions in the instructions to the user.
Ethical considerations are of particular importance when human persons are directly concerned, and therefore fairness in judgment must be secured, i.e. equity.
It will often prove useful to anticipate the difficulties that the evaluator may face in using the specific guide. Mistakes, pitfalls or drawbacks he/she may meet will be identified, and manners of overcoming them will be proposed. Examples are :
- Criteria in the specific guide are ambiguous, or difficult to interpret (because for instance the instructions for use are not clear or not complete)
- Biases : the conditions for a good evaluation are not fully satisfied
- Lack of transparency
- Conflicts of interest: they need to be prevented or corrected. Appeal to the evaluator’s ethics is sometimes needed, in order to prevent or correct such biases
- Results of evaluation are used for a purpose which is not included in the stated evaluation objectives
- “False negative” that is negative assessment of a research or a researcher, when actually it/he should be evaluated positively
- Inadequate ranking of applicants, teams, etc.
- Calls to candidates or instructions to apply for support are unclear (or at least not fully understood),
- etc.
Step 5. Actual drafting of the specific guide
5.1. Content of the specific guide
The specific guide, in its first version, should conveniently cover the following points – keeping in mind the possibility of changing the order and the importance of each section at any moment :
- A short introduction referring to a set of annexes (or links in the case of a guide put on a website). Possible items :
- Who are the intended users of the local guide (the evaluators)?
- How should the specific guide be used?
- Pitfalls, difficulties to be anticipated, and precautions to be taken to avoid them.
- A list of criteria and procedures to be followed in a clear, sequential, stepwise process, providing practical suggestions. Useful additions here can be :
- Notes about possible variations or unpredictable situations,
- Suggestion to the evaluators to be flexible, to favour clear explanations and transparency,
- Generous references to the annexes or links.
- Comments, answers to FAQ (frequently asked questions).
- Annexes or links to assist the evaluators in adequately facing potentially atypical situations.
- In some cases, when deemed fit, examples of existing evaluation tools.
5.2 Distribute responsibilities among the guide’s authors
A guide will seldom be built by an individual, and it more generally will be the work of a team. Various decisions will be made about the sharing of responsibilities and tasks among the authors of the evaluation guide : committee work, individual contributions, call to external referees, etc. Also the bases should be established for reaching a consensus among such persons.
Special attention will be given to the desirability of having an external evaluator among the users of the specific guide.
5.3 Actual writing
The stage is then set for starting to write the first version of the local guide.
5.4 Check for consistency
Return to the preliminary stages and make sure all the indications contained in this page have been given due consideration. That does not mean, obviously, that all have to be adopted.
Proceed iteratively.
Design of the specific guide’s evaluation
The specific or local guide will need to be evaluated after having been in use for a certain time, with in mind its improvement. Essentially, the questions to which such evaluation should provide answers are of the following type :
- Are the tools, the instructions, etc. clear and well understood, exempt of ambiguities?
- Are they useful (for reaching the evaluation’s objectives)? Do they allow for satisfactory decision-making?
- Do they respect the conditions of evaluation as initially stated? If yes, to what extent?
- What are the results, the achievements, thanks to using the specific guide?
- etc.
To a large extent it is desirable to design such evaluation at the time of drafting the guide, leaving the way open for later amendments.
In quite a few cases the people in charge of evaluation of development research , that is the evaluators or practitioners, while not being themselves the authors of the specific guide, may like to follow the steps listed above and critically review the guide or parts of it, in the light of their experience, local needs, changing context, etc.
Final comment: a call to the users of the present Guidelines
Whatever the type of evaluation grid or manual you are writing, please anyhow write us your comments about this chapter in order to help us improving it in the light of further experience. Thank you in advance.
