Participatory Results-based Monitoring and Evaluation (M&E)
Monitoring is about the collection of information, which will allow us to determine “where” a policy, program, or project is at any given time (and over time), when comparing to the respective targets. It provides feedback on whether an implementation is on or off track.
On the other hand, evaluation is about finding evidence of “why” targets are, or are not, being achieved. It addresses attribution and causality. It is an assessment of a planned, ongoing, or completed intervention to determine its relevance (are we doing the right thing?), efficiency (are we doing the things right?), effectiveness (are we achieving the expected results?), impact (what are the primary and secondary, positive and negative, intended or unintended, long-term effects?), and sustainability (are the benefits continuing after the policy, project, or program is completed?). It assists in the allocation of resources, in rethinking causes of problems, in identifying emerging problems, it supports finding best alternatives or discerning on competing ones, it incentivizes innovation and it contributes for reforming the public sector, and it helps building consensus on the causes of a problem and how to respond to it.
Results-based M&E differs from traditional implementation-focused M&E (inputs, activities, and outputs) as it moves the emphasis to the outcomes and goals (results) and is a continuous and reflexive process. It poses the question “so what?”, as the sole fact of achieving certain desired outputs do not necessarily mean expected results were achieved.
But aiming for outcomes may not necessarily result in sustainable solutions. In participatory M&E the processes, and all actors involved, are fundamental variables to be considered. Therefore, it tries to involve all relevant stakeholders in each stage of a development initiative, and it aims to measure and improve social transformation. The latter may be defined as empowerment derived from improved capacity, and strengthened communication skills, and from community's emotional, psychological, cultural, and spiritual needs. Development is interpreted as a complex and holistic process. Thus, some of the outcomes are intangible and long-term, which means qualitative indicators, commitment of time (in order to conquer mutual trust), and availability of resources are required. This paradigm underlines the constant need for monitoring and evaluation as an enhancer for the internal-learning process leading to self-determination, autonomy, and effectiveness. Under participatory M&E, accountability and ownership are supposed to be balanced.
M&E systems are used by the actors of an initiative as management and as organizational learning tools, assisting planning and designing a policy, program, or project, formulating and justifying budgets, implementing them, and allowing for continuous adaptation. They can also help identify unintended results, or other potentially promising practices. Finally, they provide the necessary transparency and accountability that stakeholders expect, contributing to reduce trust gaps. They are performance frameworks, living tools for implementing social transformation.
Based on the above, we could define Monitoring and evaluation as a continuous and sustainable participatory results-oriented management process that contributes to empower and better address local people’s needs, and strengthen transparency and accountability, through more relevant and efficient organizational learning processes.
Agile Methodology
Agile is an iterative results-oriented methodology, supported on human communication and feedback, allowing for requirements and solutions to constantly adapt, through the collaboration of self-organizing cross-functional teams.
Its iterative nature means it is incremental, i.e. it is done in pieces, and each adds tangible value to the whole. As an empirical methodology, it finds its strength on transparency, collaboration, inspection, and flexibility for adaptation, rather than on bureaucratic, rigid, and systematic processes. It allows for adaptative planning, early deliveries, strong accountability, and continuous improvement. It promotes values such as courage, focus, commitment, respect, and openness.
The Agile methodology is appropriate for projects of high uncertainty and complexity, where empiricism often means better results. One of the most popular frameworks that can be used for effective implementation of the empirical methodology is Scrum. This framework uses the term sprint for iteration.
A Scrum team consists of a Product Owner, a Development team, and a Scrum master. The Product Owner manages the product backlog (a prioritized single dynamic list of work, that needs to be addressed so that the project achieves its goals), is responsible for maximizing the work produced by the Development team, and is the one accountable for the project. The development team is cross-functional (or full-stack), manages the sprint backlog (list of tasks for a specific sprint to be considered done), and is the implementing body of the project. Finally, the Scrum master is the servant leader, making sure the scrum framework and values are adopted. Hence, he is supposed to assist the team in achieving its goals by clearing any obstacle. He is also mandated to advocate, among other relevant stakeholders, for the importance of Agile methodology and Scrum framework in particular.
Similarities between Participatory M&E and Agile
The traditional version of M&E tended to focus on outputs, which are not necessarily a reflection of a policy, program, or project’s success. However, a results-based M&E budgets for outputs, and manages for outcomes. Scrum’s iterative framework is all about delivering tangible products, i.e. results, after each sprint is over. Indeed, the product backlog’s priorities are aligned with the stakeholder’s expectations as the product owner is supposed to keep constant communication with them. These expectations come in the shape of outcomes: ready-to-use fully functional increments.
Under a traditional way of M&E, evaluation is often performed once, and after the termination of the initiative only. It is a waterfall approach that does not inspire flexibility, nor adaptation. Under a results-based M&E, monitoring and evaluation takes place during planning, implementation, and after the intervention is completed. These evaluations eventually lead to adaptations/improvements, and become learning material to support future endeavors. In Scrum, meetings such as the daily scrum, the sprint review, the sprint retrospective, and the product backlog refinement, are means of inspection for eventual adaptation. Whereas results-based M&E uses several different methods for data collection (e.g. field visits, focus groups interviews, questionnaires, panel surveys, and so on), Scrum appeals to face-to-face meetings. For instance, during the daily scrum meetings, questions such as what did I do yesterday, what will I do today, and what are my challenges, might be raised. The first two questions fit into the category of monitoring, as the idea is to determine where the development team is at that specific time in relation to the sprint goals. The third question targets causality. It tries to understand which obstacles are impediments for achieving the desired goals. It fits into the evaluation of the sprint. The sprint review’s main objective aligns with one of the evaluation criteria for policies, programs, or projects: relevance. In fact, this review aims to determine whether the intervention is consistent with the stakeholders’ expectations. The sprint retrospective is an evaluation of the sprint as a whole. It addresses what went well in the sprint, what could be improved, and what will be committed to be improved in the next sprint. It basically puts the focus on efficiency, and effectiveness, which are two other evaluation criteria for development initiatives. Finally, the product backlog refinement is an ongoing process of adding detail, estimates, and order to the product backlog. This step is the reflection of new learning about the product’s evolving capabilities, and how users are responding to changes. It involves transparent and open channels of communication among all stakeholders, whose voices are uniformly important. It is rather symptomatic of what participatory M&E stands for.
Participatory results-based M&E needs strong human, institutional, and management capacity. It needs integrity, honesty, and professionalism. It needs a tradition of accountability and transparency, credible and legitimate political leaders. The three pillars of Scrum are transparency, inspection and adaptation. And these assume strong human, institutional, and management capacity. Its core values include courage, focus, commitment, respect, and openness. The Scrum team have well defined roles, and accountability cannot be delegated.
Participatory methods are bottom-up processes of self-mobilization and interactive participation. In Scrum, it is the self-organizing Development team that informs the product owner how many items of the product backlog can fit in one sprint and how long it will take.
Lately, participatory initiatives in the development arena seem to be proliferating more successfully than in the 90s. They are becoming more reflexive, more realistic when involving local communities, and are making efforts to reverse some top-down trends imposed by donors. But, as they are long term, some of their outcomes are not easily observable and indicators are often hard to find. Thus, development projects should use interim outcomes, or milestones, in order to take some pressure off and please donors. These milestones could be compared to the sprint goals in Scrum, assuming that time frames are obviously different.
If we could translate the roles within participatory results-based M&E into the roles of Scrum, we would probably have something like this:
The governmental body (or other implementing institution) accountable for the implementation of a development policy, project, or program, takes the role of product owner. This entity has the responsibility to involve all stakeholders, ensure engagement and ownership, and oversee the implementation;
On the other hand, the development team may be disaggregated into two different groups:
A. The main beneficiaries (or their representatives) of a development initiative, who should actively participate in all
stages of the development initiative (such as on setting up indicators and targets for goals and outcomes);
B. And the remaining members of the implementing team (i.e. other individuals who offer any necessary expertise, but
who aren’t direct beneficiaries).
In fact, this disaggregation is justifiable, since the main beneficiaries are (or should be) the main component of the development team, but they are also the client. Indeed, the whole philosophy of a beneficial social transformation is that communities take things in their own hands. Processes become as important as results, and learning becomes an inclusive tool for empowerment. Often, to complement the development team, external experts are invited to add value to the projects;
- Finally, donors who have (at least in principle) mandates for mainstreaming core values of human dignity, and inclusive development, are the ones who lead development by funding it, and therefore take the role of the scrum master. As so, donors should always be servant leaders. They should remove any impediment for development, and make sure participatory initiatives are never overridden by top-down approaches for development.
This document focuses mainly on similarities between the two practices. However, it is important not to neglect the fact that international development is (or should be) focused on the empowerment of local people and on making this process sustainable, whereas Agile is mainly concerned with the targets set for the project. Relevance, efficiency, and effectiveness are three criteria addressed by both M&E and scrum. However, participatory results-based M&E is also interested in measuring the impact and sustainability of a development initiative, as social transformation is a complex and a long-term process.