Quality criteria for good indicators

Indicators specify the scope of the objectives in terms of coverage, scale, quantity, quality, and in terms of the time in which the achievement of the objective is expected to manifest itself. A suitable indicator needs to meet the following quality criteria and be…

Essential

The indicator measures a central or relevant aspect of the result.

Some indicators are more appropriate than others for measuring progress towards the achievement of an objective. This depends largely on the context of a project.

One must ask oneself: 

• Is this the best way to know whether the objective has been achieved?
• Is the indicator essential?
• Does it really provide the information required to make relevant decisions?
• Does it precise the change we want to achieve?

Coherent

The indicator directly connects the changes with the outputs that the project can generate.

A coherent indicator should represent changes at each relevant level of change regarding: 

• the well-being of people and the environment, 
• the performance of households, organisations and institutions,
• the behaviours, attitudes, routines and practices of individuals and groups  

Coherence also implies being specific, i.e., that the indicator corresponds to information directly associated with the objective or result to be observed and that allows the observer to verify whether or not the project has or has not met the objective.

The indicators detail aspects such as to what extent, with what specifications and in what place (reduction of domestic violence index in the department of Jinotega, mileage of coffee roads repaired, hectares of land under organic production in the territory of Asturias, x amount allocated for the Road Maintenance Fund, % extension of drinking water in region X, etc.) the change is taking place.

Coherence in terms of hierarchy or coverage refers to the fact that the indicators correspond to the same scope of action (field, farm, community, watershed, municipality, etc.) of the proposed change. If the result is at the farm level, the indicator should be at the farm level. It would be inconsistent to present a national level indicator in this example.

The indicator also needs to be sensitive to short-, medium- or long-term changes.

Coherence also includes that the indicator needs to be independent, i.e. there can be no cause-effect relationship between the indicator and the objective, because then we would be talking about two different levels of change.

Realistic

The indicator specifies a highly probable effect.

The scope of the indicator needs to be achievable and feasible.

It refers to the likelihood of achieving the indicator, that means the change, in all its aspects.

The achievability of an indicator depends on the context of a project.

Often the optimum cannot be reached and a more realistic target has to be sought (e.g. immunization of all children under 5 years of age).

Verifiable

The indicator is verifiable, i.e. the necessary information can be collected and objectively measured.

Here it is important to ask the question whether the project or its participants have the means, knowledge and time to observe the indicator, i.e. whether its measurement is feasible.

Two people measuring the same indicator should always be able to collect the relevant information in the same way, using clear common criteria.  This information should be relatively easy to collect. Indicators can be measurable quantitatively or qualitatively.

If we are going to use a percentage, we must be clear about the reference data, e.g. from a baseline.

Once we are going to evaluate the degree of accomplishment of our project, it might make a huge difference if we established an indicator that refers to: 70% from 200 farmers o 70% from 5000 farmers. If we only say 70% from farmers, the reference data – number of farmers addressed by the project –  is not clear.

Acceptable

The costs of data collection, processing and analysis are in an acceptable relationship to project costs.

If monitoring an indicator is too costly, then it is not acceptable.

For example:

If we want to measure the reduction of sedimentation of the Old River and we would have to hire specialists with equipment that does not exist in the country, it is more appropriate to rethink the indicator.

In one child labour reduction project, the verification of an indicator related to the removal of child labour involved monthly data collection and processing for each child. 

In the end, the cost and time required were equal to or even greater than the time invested in child labour reduction strategies (awareness raising with parents, training for teachers, meetings with coffee producers…), so either the formulation of the indicator needs to be changed or the methodology and verification instruments need to be revised.

Neutral

The indicator itself should not be value-loaded, it should only “indicate” the existence or not of the change and its magnitude.  The assessment of the change, whether it is positive or negative, whether it corresponds to what we expected or not, arises as a product of the analysis of the indicators.

The indicator is like a metre that tells us, for example, that a wall is 20 metres long, but it does not tell us whether that 20 metres is a lot or a little. That assessment should be given by the analysis made either by confronting that indicator data (20 metres) with the planned amount to see if more or less than planned has been done or by comparing it with the average work of a bricklayer.

For example: 

Instead of stating “growth of self-esteem” it is more appropriate to place an indicator that measures “the degree of self-esteem”. Only by measuring growth we would not be able to know how this change in self-esteem is manifesting itself or its magnitude.

In these cases, it is very important to be able to define scales that will specify the “levels” or “degrees” of change.

“# families have established 100 lm (linear meters) of living barriers on their farm”.

If 100 lm of living barriers have been established on a farm, we do not know if it is too much or too little until we put the data in relation to a reference value.

If, for example, 100 lm on a hectare with x degrees of slope is common, then we could say that this is a good achievement.

If the 100 lm, however, refers to a farm of 10 hectares, it is probably not very meaningful. The indicator itself does not give us this information, but the analysis, the comparison, the cross-checking with other reference values.

Remember the principles identified by Hans Rosling.

Stakeholder-oriented

Is the indicator meaningful, understandable and transparent for relevant stakeholders (youth, producers, politicians, etc.)?

Are there local indicators that can be used?

Gender equity oriented

Does the indicator reveal specific knowledge and issues by differentiating changes in women and men?

Sustainability-oriented

Do the selected indicators represent all dimensions of sustainability? (cultural, politico-institutional, social, economic, ecological)