What we learned: SenseMaker as a Monitoring and Evaluation Tool

This project is funded through the HIF’s “Exploring New Ways to Improve Monitoring and Evaluation of Gender-Based Violence Interventions in Humanitarian Settings” Challenge. It arose out of recognition that data about the quality and longer-term impact of sexual and gender-based violence services is often lacking.

Our team used Cognitive Edge’s SenseMaker® as a monitoring and evaluation (M&E) tool for gender-based violence (GBV) services in Lebanon. SenseMaker® is a mixed qualitative/quantitative data collection tool used to empower individuals to anonymously record and interpret their own stories about their experiences of accessing a GBV service. In collecting these stories, we aimed to examine the feasibility and added value of using SenseMaker® to generate timely mixed methods results and a more nuanced understanding about experiences accessing GBV programmes.

Project partners included:

• ABAAD Resource Center for Gender Equality
• International Rescue Committee (IRC) Lebanon
• Queen’s University
• United Nations Population Fund (UNFPA)

With data collected by:

• ABAAD
• Akaar Network for Development (AND)
• Caritas Lebanon
• International Medical Corps (IMC) Lebanon
• IRC Lebanon
• United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNWRA).

After pilot testing and refining the survey earlier in 2018, these six GBV service providers asked women and girls attending their programs across five different sites in Lebanon (Beirut/Mount Lebanon, Tripoli, Akkar, Beqaa, South Lebanon) to complete the SenseMaker survey. Over a ten-week period, 198 self-interpreted stories were collected.

Data was reviewed at a two-day closing workshop in Beirut in September 2018 where partners also engaged in deliberative dialogue about the feasibility and value of using SenseMaker® as an M&E tool for GBV programmes. The project’s key learning points are highlighted below:

1.Existing M&E Gaps Identified by Gender-Based Violence Service Providers in Lebanon

Across organisations, there was more quantitative M&E data collection than qualitative and most organisations were more satisfied with their monitoring. There was a recognised need for more impact evaluations, for additional qualitative data and also for more holistic M&E approaches.

2. Logistics of Collecting SenseMaker® M&E Data

Given that SenseMaker® was new to most team members and that literacy skills/experience with technology were limited among most respondents, the survey had to be facilitated by a staff member for a majority of participants. This required a significant time commitment by the team. The lack of dedicated staff training prior to data collection was also identified as having led to some unease with the survey. Typing the stories into a browser was time-consuming and many of the women and girls were uncomfortable having their voices recorded using the SenseMaker app on a tablet. And finally, for some organisations, the availability of staff to facilitate the survey was a challenge.

3. SenseMaker® M&E Data

The dyad questions (where participants were asked to plot their perspectives along a continuum between two extremes) were identified as being poorly understood by a majority of participants and would need to be refined if they were to be used in the future. There was consensus that SenseMaker® provided an improved understanding of how programs were perceived and better captured the needs of women/girls in comparison to existing M&E tools. There was an appreciation for the mixed methods approach and for having qualitative data to contextualise the quantitative data.

4. Additional Added Value of SenseMaker® M&E Data in Comparison to Existing M&E Approaches

SenseMaker® was believed to have provided a unique combination of macro and micro data (big picture combined with individual experiences) and was perceived to have filled an existing gap around the lack of qualitative data for program evaluation. There was a sense that SenseMaker® would have more utility in acute humanitarian settings or at the onset of crises since SenseMaker® data can be more readily available. Two additional uses of SenseMaker® were recommended for consideration in the future: a) to track program perceptions longitudinally, and b) as a tool to better understand nuances around the root causes of GBV in particular contexts. Partners were in favour of the mixed methods approach and noted that the narratives would be useful for advocacy purposes.

5. Scalability of SenseMaker® as an M&E Tool

The biggest perceived barrier was cost, financial as well as human resources. However, if multiple service providers were to use a shared SenseMaker® M&E tool, the associated fixed cost would be divided between the organisations. Since Android tablets are more widely available and less costly, future SenseMaker® M&E tools should be configured for Android operating systems. Scale up of a SenseMaker M&E tool would require commitment and engagement both from the individuals responsible for collecting and using the data as well as from the organisations’ management.

Conclusion

Despite the challenges identified throughout this feasibility pilot, our team concluded that SenseMaker® had yielded some unique insights and its mixed methods approach provided a more comprehensive understanding of women and girls’ experiences accessing programmes. SenseMaker® revealed important new data about who was, and who was not, accessing services.

In summary, we believe that SenseMaker® is feasible and could have added-value as an M&E tool for GBV programs particularly in more acute humanitarian settings. As an M&E tool, SenseMaker® is better suited for women and girls with the literacy and technological skills to complete the survey independently. To successfully implement SenseMaker® as an M&E tool, financial and human resource support would have to be available and it would have to be thoughtfully integrated into existing M&E activities.

The project’s full report can be downloaded here.

Additional information is available by contacting the project lead, Susan Bartels at [email protected]

Photo Caption: Final workshop held in Beirut in September 2018. Credit: Luissa Vahedi.