Comparing apples to oranges: Measuring different methodologies in different contexts

The challenge of monitoring Community Engagement

Our Landscape Review found that systematic and documented field monitoring is notoriously patchy in the humanitarian sector; with community engagement being particularly challenging. Logical frameworks may not clarify whether the hoped-for result of community engagement is ‘instrumental’ in obtaining information about community sanitation preferences and needs in order to design more appropriate facilities, or whether the process is aiming for more complex goals such as ownership and empowerment. Since this is what we were aiming to do – monitor if community engagement approaches were successful in learning user preferences so that facilities would be user-centred, we realised in working with the three other partners who are implementing different community engagement methodologies in five locations, that it felt as though we were comparing apples to oranges!

One size does not fit all

The partners are implementing their community engagement methodologies across very different contexts; from rapid on-set emergencies in Bangladesh and Uganda, to protracted crises in Iraq and Lebanon. In the former, they may have only recently started working due to the nature of a rapid onset emergency, whilst in the latter, they have been present for some time as is the case with a protracted emergency. In the development of our M&E logframe, we had to consider all these variations while ensuring that comparisons to be drawn across all project locations.

We put our heads together and decided that it would make sense to create a table where we could align their different methodologies against the stages of project implementation: preparation, assessment, design, pilot-test, feedback and M&E, adaptation, completion of final model, and further feedback.

We sent this table around to our partners and under each stage of implementation, information was requested for various aspects of the phase such as: methodology planned, a summary of activities, estimated timelines of the various project phases, tools to be used, teams involved and any concerns or risks they may envisage during implementation. This also served as a way in which Oxfam, as R&E partner, could look at the methodologies from another vantage point – allowing us to match similarities and identify differences across their unique and innovative methodologies. It also felt like we were no longer comparing apples to oranges and in possession of a completed Cumulative Methodology table, could complete the M&E logframe.

A usable M&E logframe for user-centred sanitation design:

The completed M&E logframe includes three key outcomes with 14 required indicators across all three outcomes. The three key outcomes are:

1. Community members feel more comfortable using maintained latrines; with indicators measuring various aspects of latrine use and maintenance
2. Community members are satisfied with latrines; with indicators measuring community satisfaction with all aspects of the latrines ranging from siting and design to maintenance
3. Evidence of the value of community engagement in sanitation during emergencies is generated: with indicators measuring the level of community participation in the process of designing the latrines

While the logframe facilitates the monitoring of the methodologies undertaken with respect to those three aspects of use and maintenance, community satisfaction and their participation in the process, it does not enable us to evaluate whether greater community engagement (including empowerment, trust and mutual respect) leads to improved latrine construction that is timely, appropriate, consistently used and community-owned.

What’s next?

We are currently working to develop an evaluation protocol and a set of standard evaluation tools with the support our academic partner London School of Hygiene and Tropical Medicine (LSHTM) to explore our hypothesis that greater community engagement leads to improved latrine construction and usage.