Introduction indicators

What is it?

For a pdf file with all info and examples click here

A quick search on the world-wide-web on "indicators" provides us with an overwhelming amount of hits on business indicators, economic indicators, social indicators, environmental indicators, health indicators, education indicators, governance indicators, quality of life indicators, etc. etc.

All these indicators have one thing in common; they refer to specific information. Since managers regularly require specific information to enable proper decision-making, indicators often play an important role in profit as well as non-profit organisations.

This document intends to provide more insight in indicators, especially in a development co-operation context. It explains the concept, highlights different aspects, gives suggestions on how to formulate indicators and provides several examples of indicators from recent real-life development co-operation efforts at project, programme as well as policy level.

Definition

The English Language Dictionary describes an indicator as "an instrument which gives you information". In line with this description, indicators come into the picture in a development co-operation context at the moment that specific information is required. In this context, different definitions are being used.

  • According to OECD/DAC, an indicator is: "A quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect changes connected to an intervention, or to help assess the performance of a development actor" (DAC Glossary of Key Terms in Evaluation, May 2002)
  • According to the definition adopted by USAID, an indicator is: "a variable, which purpose it is to measure change in a phenomena or process"
  • The European Commission describes (planning) indicators as: "a description of the project's objectives in terms of quantity, quality, target group(s), time and place"

Although, differences can partly be linked to the moment indicators come into the picture (stages in the project cycle), these definitions also illustrate two existing views on indicators. While in the American interpretation the indicator equals the "variable", a European interpretation often results in including also a timeframe and a base -and target value to the variable. In this last interpretation, the indicator is (also) used to elaborate the objectives further, which are still formulated in rather general terms. In this syllabus you will find examples of both interpretations.

Indicators in the project cycle

Indicators are used at several stages in the project or programme cycle (see figure 1). Since characteristics of these stages differ, the functions of indicators may also differ.

During the identification stage the plan for the project or programme is being formulated in general terms and, normally, (planning) indicators are not yet formulated at this stage. Nevertheless, indicators may already play a crucial role in getting more precise information on the context of the intervention and on the problems to be tackled. During this identification stage, organisations will try to reinforce the quality of the situational analysis by using specific indicators.

For example, before starting an HIV/AIDS intervention, indicators on HIV prevalence may be helpful to compare the situation between regions/areas and/or groups of beneficiaries and justify a regional or target-group focus. Moreover, once the intervention is being implemented, these data may well serve as important base-line information.

For HIV/AIDS related indicators, important efforts are being undertaken by UNAIDS ( www.unaids.org). For more general health related indicators consult for example WHO ( www.who.int).

During the formulation stage, the initial plan needs to be worked out into more detail and, generally, several indicators are now being formulated. At this stage, the indicators are called planning indicators or, sometimes, also Objectively Verifiable Indicators (OVI's). Objectively verifiable refers to the fact that different persons will come up with the same information while using the indicator.

In case the plan is worked out in a logical framework matrix, the planning indicators or OVI's are presented in the second column of this matrix.

Indicators are generally formulated for the Overall Objective(s), the Project Purpose and for the Results. This provides sufficient level of detail for the formulation stage.

Together with the indicators, the sources where to find the information referred to in the indicator, are also identified. These so-called Sources of Verification (SoV) are included in the logical framework matrix in the third column. Identification of these SoVs at this stage is highly recommended, since discussions on where to find the information or how to collect it, often lead to reformulation of the indicator. In the worst case that the information referred to in the indicator cannot be obtained, the indicator becomes useless and a new one should be formulated.

The development of indicators during the formulation stage is crucial. First of all, when objectives are still vague, (planning) indicators are necessary to obtain information about target group, timeframe and baseline and target values (European approach, see page 2). Without this information, resource allocation and budgeting as well as operational planning remain almost impossible. For example, an objective like "increased worker productivity" means little without specifying the exact nature of productivity, and with how much it is supposed to increase for how many workers and within which timeframe. In this case, the formulation of the indicators is even a precondition to enable proper finalisation of the plan.

Second, it is also highly necessary already at this (formulation) stage to define how to track progress towards the objectives during implementation. See for example the indicators formulated to track progress towards the achievement of the Millennium Development Goals. Since in this case, timeframe and target value are already clearly defined in the so-called "targets", the formulation of the indicator is limited to the variable (American interpretation).

As explained above, a complete plan will include information on target group, timeframe, baseline and target values. This information can be included already in the objectives or is further specified in the indicators. The completed plan, including the indicators, will now provide a solid basis for the management during implementation and thus also for monitoring and evaluation.

During the implementation stage, monitoring and evaluation activities provide managers with particular information, for which, again, indicators are used. Obviously, the (planning-) indicators formulated at the level of results, purpose and overall goal, will be the starting point for monitoring and evaluation. However, during implementation, managers will also be interested in other aspects of the intervention, like depletion of budget or specific bottlenecks caused by external factors. In this context, some organisations distinguish input indicators, output indicators, etc. referring basically to the type of monitoring they are used for (see below)

Different indicators and focus of monitoring

Moreover, a manager might want to monitor management aspects that are not even mentioned in the logical framework (like HRM issues, reporting frequency, etc.). Based on the specific information needs of managers, a variety of monitoring indicators can be identified.

For evaluation and, as a consequence, also for indicators for evaluation, the same principal is applicable. Depending on the evaluation criteria (like relevance, effectiveness, efficiency, sustainability, and impact) the manager's information needs (subject of analysis) and related to that the focus of the evaluation indicators will have to be further defined. Again, planning (and monitoring) indicators will be a good basis for evaluation activities. For example, since impact is closely related to achievements at overall goal level, the changes in the different variables (planning/monitoring indicators at that level) will facilitate the appreciation of impact. Similarly, planning/monitoring indicators at purpose and result level will provide information to evaluate effectiveness.

Monitoring indicators on budget depletion and on use of resources will facilitate the evaluation of efficiency and indicators related to context may well support appreciation of sustainability.

Type of indicators

We distinguish two types of indicators:

1) Direct indicators, which refer directly to the subject they have been developed for

2) Indirect indicators, which only refer in an indirect way to the subject

Direct indicators

These indicators directly pinpoint at the subject of interest. This is often the case with operational and more technical subjects. What the manager wants to know, can be (and generally is) measured directly. A good example of a direct indicator, which might not be so easy to measure, is "the proportion of the population below $ 1 per day" (See MDG 1 presented above).

Indirect indicators

Indirect indicators (or proxy-indicators) refer in an indirect way to the subject of interest. There can be several reasons to formulate indirect indicators:

  • The subject of interest cannot be measured directly. This is particularly the case for more qualitative subjects, like behavioural change, living conditions, good governance, etc.;
  • The subject of analysis can be measured directly, but it is too sensitive to do so, for example level of income or, in the context of an HIV/AIDS intervention, "safe sex";
  • The use of an indirect indicator can be more cost-effective than the use of a direct one. As such, indirect indicators are very typical management tools. Generally, managers are not looking for scientifically reliable data but for management information. An indirect indicator may very well represent the right balance between level of reliability of information and the efforts needed to obtain the data.

Acronyms SMART & SPICED

More and more organisations are called by the general public and by their donors to account for their achievements in terms of concrete results. In order to make this possible for development interventions, projects as well as programmes are expected to become more and more "SMART":

  • S Specific
  • M Measurable
  • A Achievable
  • R Relevant
  • T Time-bound

In this context, some organisations put emphasis on the formulation of "SMART" objectives; others focus on "SMART" indicators. In the first case, since objectives already include a timeframe and baseline and target values, often indicators are formulated as variable (again, see also section 2). In the second interpretation, the variable is completed with a timeframe, a baseline and target values. Consequently, these indicators are more specific, including information about target groups and what needs to be achieved for these target groups (SMART indicators).

Obviously, "SMART" objectives and/or indicators play an important role in result-based management and in the discussion on accountability. When to become SMART, during the formulation of the intervention logic (activitiesŪresultsŪobjectivesŪgoals) or while formulating the indicators, is of less importance.

Roche (2002) claims that when indicators are used more as specific examples of change (Impact Assessment for Development Agencies), different characteristics become important. In this context, he refers to SPICED indicators:

Properties

Definition

Subjective

Informants have a special position or experience that gives them unique insights which may yield a very high return on the investigators time. In this sense, what may be seen by others as 'anecdotal' becomes critical data because of the source’s value.

Participatory

Indicators should be developed together with those best placed to assess them. This means involving a project's ultimate beneficiaries, but it can also mean involving local staff and other stakeholders.

Interpreted and communicable

Locally defined indicators may not mean much to other stakeholders, so they often need to be explained.

Cross-checked and compared

The validity of assessment needs to be cross-checked, by comparing different indicators and progress, and by using different informants, methods, and researchers.

Empowering

The process of setting and assessing indicators should be empowering in itself and allow groups and individuals to reflect critically on their changing situation.

Diverse and disaggregated

There should be a deliberate effort to seek out different indicators from a range of groups, especially men and women. This information needs to be recorded in such a way that these differences can be assessed over time.