ILEIA Newsletter • 12 nº 3 • December 1996
Agreeing on indicators
AS-PTA is a Brazilian NGO working in partnership with farmers, associations, and rural trade unions (STR) of the Northeast, to seek more agro-ecologically and economically sound forms of agricultural development. As can be said of many NGOs, despite the intense and dedicated work of AS-PTA members, there is little systematic, documented evidence on the impact of their efforts. Unfortunately, lack of proof is often interpreted as a lack of success or an absence of evaluation. What to monitor and what indicators to use are among the first steps to be agreed upon in a partnership between farmer organisations and NGOs jointly seeking to track the impact of their work. The following experience tells one story of how to agree on indicators that are meaningful to both farmers and NGOs.
Photo: Irene Guijt
Projeto Paraiba
Within a small NGO like AS-PTA, the joint challenge of environmental regeneration and economic viability of smallholder production has forced a clear prioritisation of activities. Community-based seed banks to overcome the lack of timely access to seeds, contour planting for dealing with soil degradation, pigeon pea intercropping for fodder in the dry season, and biological pest management in cash-generating banana plantations are, today, key extension activities. Also ongoing are a series of farmer experiments, covering different themes and ranging from ant control to green manuring of bananas. Yet other activities reinforce the institutional capacity of the partners, and deal with local and regional networking.Since 1993, AS-PTA has been working with smallholders in two municipalities of the Agreste: Solanea and Remigio. The partnership, Projeto Paraiba, involves three levels of farmer participation:
• A nucleus of about ten farmers, affiliated with the STR, in charge
of strategic planning, data analysis, monitoring and evaluation. This group
is also responsible for most dissemination and monitoring work in the field.
• The second level of farmer participation involves between 60 and 80
farmers, men and women. It includes community association leaders and individual
farmers engaged in joint experimentation. Practically all are also involved
in key moments of monitoring, evaluation and planning, particularly those related
to the experiments.
• The third level includes activity-specific collaboration with the general
farming `public’ and community associations, covering 30 communities and
over 500 farmers, who are keen to adopt particular measures.
• The big question now, and one faced by many similar organisations, is
to track the changes that are brought about as a direct result of these activities.
The need for monitoring
The lack of data is clearly a problem for long-term planning of the project itself and for accountability to donors. It also limits the scope for influencing the debate and spread of alternative sustainable agriculture practices at both a local and national level. For Brazil to shift to an alternative model for agricultural development, dissemination of alternatives such as those suggested by 'Projeto Paraiba' is essential. Proof that such alternatives work not only focuses attention on the concrete impact of agricultural practices, but on the processes that allow their development and implementation - namely, the slow and close work and cooperation between rural trade unions, community associations and farmer groups.AS-PTA’s monitoring approach to date has been largely 'crisis-driven', activated when staff have identified an information gap that needs quickly filling. Basic monitoring and evaluation forms for farmers and STR representatives to evaluate the rate of adoption of certain activities are deployed with varying success. The data thus collected focuses on the number of farmers, workshops and experiments; on the area under the innovation; on the number of seed banks; etc. Yield data are also being collected but accurate recording is proving difficult. Evaluation of the impact of the work is complex since one farmer may use several new technologies at once. And there is no agricultural baseline data in the region, further complicating attempts to establish what the direct impact of 'Projeto Paraiba' has been.
Longer term collaboration is under way to develop, with the STRs and farmer interest groups, a participatory monitoring system to allow the collection and processing of more useful information with less effort. The venture aims to provide i) an ongoing learning experience that can help strengthen group structures and improve the planning process and effectiveness of interventions; and ii) data to fulfil accountability criteria of donors and to support local and national level policy efforts of the STR, AS-PTA and Rede PTA (a network of 23 NGOs).
Putting Theory into Practice
The first steps were taken with workshops in January and July 1996 with farmers, STR representatives, and other affiliates. These helped identify:• the objectives of the partnership viewed from the perspectives of AS-PTA
and two STR groups, plus prioritising the objectives for monitoring purposes;
• the most useful indicators for tracking progress, i.e. the information
needed to assess whether objectives are being achieved;
• the best methods for collecting and recording information, with several
innovations aimed at adapting existing methods to suit the indicators identified
and the local cultural context.
Photo: Irene Guijt
For each of these activities, AS-PTA and the Remigio and Solanea STRs formulated their own hierarchies of objectives, representing short-term activities, medium-term results, and long-term goals. These were then combined in further discussion to develop a single 'objectives tree’ per activity, containing more than a dozen objectives per activity. Within each `tree’, several objectives were prioritised, since it was not possible to monitor all of them. A range of qualitative and quantitative indicators were then formulated for each prioritised objective. Methods were then developed to assess the indicators, drawing on both conventional and more participatory methodologies, such as PRA.
Steps now under way include further adjustment of the methods identified, data collection and analysis, monitoring methodology, and dissemination of the latter to other NGOs and trade unions in Brazil. The existing databank is being modified radically to accommodate new information needs and to remove data that serve no clear purpose.
Besides this process, which has taken place largely with STR representatives and affiliates and AS-PTA staff, monitoring is also being pursued with an important third stakeholder - the farmer experimentation groups. Discussions are starting within each group to obtain farmers’ perspectives on: the objectives of experiments; indicators that farmers need for assessing the value of the intended change to the agricultural production system; and appropriate monitoring methods.
The whole process is also expected to serve other purposes, such as building the capacity of farmers to monitor and thus contribute to technology development, creating sustainable local experimentation groups - a rare feature in rural Brazil - and providing data that is useful in evaluation and extension. Documentation of the results of the process in both Paraiba and Minas Gerais is being disseminated throughout the Rede PTA. A final regional workshop is also planned to share the methodology and data with policy makers, scientists, and other farming groups.
Lessons learnt
Spend sufficient time on the objectives. If these are not clear, it is impossible to identify indicators to monitor them. During the January workshop, creating `trees’ to distinguish between short-, middle- and long-term objectives revealed that, contrary to expectations, none of the partners were completely clear about the objectives of `Projeto Paraiba’. The cause-effect linkages identified were too simplistic. For example, it was thought that contour planting would lead directly to the objective of `diminishing migration to cities’, while, clearly, there are many other factors that contribute to migration. A further three meetings were thus needed. Another two meetings were called to share the results of these discussions and to agree on a common agenda. The process was lengthy though it might not take as long for more established partnerships.Know the end use of the data collected. Initially, the focus was on the direct measurement of biophysical properties. Yet the degree of accuracy needed is often less than assumed, especially if the data are to be used for general planning or for sensitising farmers. For example, one of the main objectives of contour planting was `soil and water conservation’. Indicators initially identified for measuring it were soil retention, moisture retention, and organic matter content. But these are impossible to measure for an understaffed and underfunded NGO such as like AS-PTA.
When discussing for whom and for what purpose the information was intended, it was realised that such precise data were unnecessary. The information was to be used in farmer-to-farmer extension and donor reporting, both of which (in the case of Projeto Paraiba) did not require scientifically valid data. Exact organic matter content was less important than knowing, for example, that six out of ten farmers had noticed a significant change in soil humidity as a result of planting along contour lines. The indicators were duly condensed into one: the frequency with which positive and negative changes were noted by farmers planting along contour lines.
Work collectively, slowly and realistically. If monitoring is to be part of a sustained learning process it has to have local relevance and be feasible in the long run. Ensuring long- term monitoring by farmers and STR has meant involving them in every stage of the design. This has taken more time than is usually the case in developing a monitoring system. It has also meant compromises. For example, the temptation to measure everything in order to provide proof for hard-headed scientists, has been firmly resisted. As Paula, one of the agronomists at AS-PTA said during the June workshop: "I want the monitoring to continue independently of us. What use is it to chose indicators and complicated expensive methods that will be dropped as soon as we pull out?"
Allow for continual change to the system. Few of the indicators chosen will retain value over a long period, particularly for experimentation groups. As activities are completed or adjusted, objectives will change. External factors will also influence objectives and require the updating of both indicators and methods. Given that such continual change of the monitoring process will be necessary, it is even more important to build local capacity. Sustainability of monitoring will hinge on the farmers knowing when and how to adapt the system.
Allow for different levels of farmer participation. Individual farmers, farmer organisations and NGOs do not have the same monitoring needs and interests. All are plagued by limited time. Choices must therefore be made as to who to involve and when. In our case, for example, the uptake rate of contour planting is crucial information for AS-PTA and STRs, but perhaps less so for the individual farmer. But when it comes to tackling ant-inflicted damage, farmers, STR and AS-PTA will all want to know the effectiveness of proposed measures. These differences led us to organise separate discussions - one involving the STRs and AS-PTA, and the other involving both of these and the experimenting farmers.
Unresolved queries
Decreasing Unsustainability = Increasing Sustainability? We are clearly not assessing the overall sustainability of the alternative agricultural approach for smallholder production being developed by Projeto Paraiba. By focusing on change as a result of a limited number of agricultural innovations, the information generated will help assess decreases in the unsustainability of smallholder production.For example, AS-PTA knows that the lack of timely access to seed can have a disastrous impact on yields. Providing seed through seed banks will thus lead to a decrease in this particular constraint, and a decrease in unsustainable production. Data on the number, use, and quality of seed banks should constitute enough `proof’ for donors and farmers that progress towards this is being made. However, whether it represents enough data for policy makers to agree to widespread support for these banks is less clear. And it does not constitute proof that `agriculture is sustainable'.
Are we measuring the `right’ indicators? We have consciously left out much that could, in theory, be monitored. In particular, we have left out higher-order impacts, such as sustained improvements in well-being and reduced migration to cities. The higher the objective, the more tenuous the cause-effect linkages become. Have the seed banks really had an impact at that level? What about external factors, or the impact of other activities? Impacts at such levels will almost certainly be a result of a variety of factors, including that of `Projeto Paraiba' itself. The question we have yet to answer is: are we monitoring at the right level and is it feasible to monitor at higher levels? For monitoring higher order objectives, baseline data become essential: data on well-being, on soil properties, on income levels, etc. Accurate data on this requires a mammoth effort of time and money, and is not within the means of Projeto Paraiba.
How to deal with external influences? Many monitoring processes simplify causal linkages, particularly those related to farmer participation. For example, an enormous growth in farmers planting fennel without pesticides could be registered two years after the intervention of AS-PTA Nordeste. But this coincided with a new rural credit scheme for the cultivation of traditional cash crops such as fennel. Isolating the impact of AS-PTA's efforts as compared with the influence of the new subsidy becomes a tricky matter. We can only chart significant external influences and attempt to understand the relative weight of their impact.
For whom is the data intended? Bringing together various stakeholder groups - farmers, STR and AS-PTA - means merging different world views, priorities and capacities. Both AS-PTA and STR identified many higher order objectives, such as `strengthening social organisation' and `financial independence of smallholders'. But in the workshops differences emerged in the setting of priorities. To ease the decision-making process, AS-PTA tended to let STR priorities determine the final choice. But it must now continue to monitor its own priorities alongside the shared priorities, some of which will require direct biophysical measurement as proof for the scientific world.
This begs the question of the compatibility of participation and science. Of what interest is it to farmers and the STR to move science along? To what extent can AS-PTA ask them to monitor indicators that are not their priority? Who will foot the bill? This picture would become even more complicated if more partners were involved, such as municipal councils, university researchers, government agricultural agencies, etc.
The quest for a sustainable tracking of change has, in our case, only just begun. But we have moved beyond the `crisis-driven' situation of before. The effort to make objectives and choices more explicit has been an important step. We hope to report soon not only about the monitoring system but also about the impact of `Projeto Paraiba'.
Irene Guijt, Sustainable Agriculture Programme, International Institute for Environment and Development (IIED), 3 Endsleigh St. London WC1H 0DD, UK
Pablo Sidersky, AS-PTA, Ave. Conda de Boa Vista 1295, Loja 8, Recife, Pernambuco, 50060-003 Brazil. Tel/fax +55-81-4213610. Email: asptane@ax.apc.org
Print this