General
What is called here "other dimensions" are dimensions which, at times, may be important in a specific situation, but as a rule are not assessed in the majority of cases.
While relevance, scientific quality, valorisation, performance, impact, comprehensiveness - for example - are used in most evaluations, the "other dimensions" are either too exceptional or too much ill-defined to be described in the present Guidelines. Other dimensions are indeed described in the Guidelines, but are not very commonly used, such as environmental implications, sustainability, or cost-effectiveness. Others could be, depending upon the case:
- contribution to the achievements of the Millenium Development Goals (example: the Belgian Prize for Development Cooperation),
- added value to other initiatives (from the same sponsor or from other organisations),
- the structure of evaluation, etc.
The present Guidelines do not attempt to give excessive importance to a classification of all possible dimensions. Their aim is to provide the specific guides' authors with suggestions and options. The author of a specific guide may indeed feel necessary, or be requested by the evaluation sponsor, to include particular criteria or questions in order to meet special needs. He/she may also, if necessary, consult the examples provided in the annex.
Here below are presented some dimensions that might be relevant in specific circumstances.
Validity
Validity generally refers to the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world. Validity of a measurement tool is considered to be the degree to which the tool actually measures what it claims to measure.
Ecological validity is the extent to which research results can be applied to real life situations outside of research settings. This issue covers the question of to what degree experimental findings mirror what can be observed in the real world (ecology = the science of interaction between living organisms and their environments ). To be ecologically valid, the methods, materials and setting of a research must approximate the real-life situation that is under investigation. Unlike internal and external validity, ecological validity is not necessary to the overall validity of a study. http://en.wikipedia.org/wiki/Validity_(statistics)
Reliability
Reliability refers to the consistency of a measure or concept. The following are three prominent factors involved when considering whether a measure is reliable :
- Stability : this consideration entails asking whether a measure is stable over time, so that we can be confident that the results relating to that measure for a sample of respondents do not fluctuate. This means that, if we administer a measure to a group and then administer it again, there will be little variation over time in the results obtained.
- Internal reliability : the key issue is whether the indicators that make up the scale or index are consistent - in other words whether respondents' scores on any one indicator tend to be related to their scores on the other indicators.
- Inter-observer consistency : when a great deal of subjective judgment is involved in such activities as the recording of observations or the translation of data into categories and where more than one observer is involved in such activities, there is the possibility of a lack of consistency in their decisions. This can arise in a number of contexts, for example : in content analysis where decisions have to be made about how to categorize media items; when answers to open-ended questions have to be categorized; or in structured observation when observers have to decide how to classify subjects' behaviour. (Bryman, 2004)
References
|
