Evaluations: Actor Based Change - Iraq
BACKGROUND
Artios provides Monitoring, Reporting, Evaluation and Learning (MREL) services for Energy Transition, National Security, and Mine Action activities.
From November 2022 to March 2024, Artios collaborated with Danish Church Aid (DCA) and IHSCO in Iraq on a proof-of-concept project for the United Nations Mine Action Service (UNMAS) Iraq . Artios presented the learnings as part of an expert panel on ‘Localisation’, at the NDM conference in Geneva in May 2024.
OUR ROLE
Artios' evaluations followed OECD-DAC criteria and used Outcome Mapping, employing mixed methods: 1) Desk reviews, 2) Primary research through participatory observation, focus groups, workshops, key informant interviews, and 3) Digital surveys. We ensured linguistic, technical, and cultural appropriateness by pairing an international Subject Matter Experts with Iraqi co-evaluators. Our evaluations combined remote interviews and site visits in Baghdad and Erbil. Our internal quality management system upheld research benchmarks, ensuring objective, reliable, and valid results.
RESULTS
Fig. 1. Dimensions of Capacity Development
The initial evaluation found it challenging to determine repeatable and scalable methods. It highlighted successful and unsuccessful examples and offered recommendations for 2023/24 development. For instance, capacity development should consider localisation and its impact across three dimensions: individuals, organisations, and the enabling environment (see Fig. 1). In Iraq, all three dimensions require significant work for effective localisation, with a conducive enabling environment (policy/political/economic) being crucial for sustainable capacity development.
Due to the project's basic MEL framework, traditional methods were found to be insufficient. Artios used outcome mapping to trace activities and impacts, identifying significant patterns and themes. The project emphasised balancing judgment—allowing autonomy for problem-solving— and control— strict adherence to procedures. This balance was challenging in Iraq's unpredictable environment and under pressure to achieve quick results and hit preset targets.
A key recommendation put forth based on lessons learnt was to use the MEL framework more effectively, to influence controllable aspects and understand project benefits (see Fig. 2). This framework helps identify available evidence and appropriate communication methods for each stakeholder. It is crucial to manage expectations about the resolution and confidence required by each stakeholder, as the degree of accuracy and confidence decreases outwards, from the sphere of control towards the sphere of interest. Projects often focus too much on the outer sphere of interest, which, while important, is usually out of scope and should be handled by other stakeholders.
Fig. 2. Spheres of Control, Influence, Interest
The evaluation showed significant relational and technical growth, driven by relationship building, experienced operational staff, and strategic hiring. Communication and persistence were found to be key success factors, more so than technical intervention.
Figures and photos adapted from and credited to :1. https://rebelwithcausation.com/actor-based-change-frameworks/2. OECD Capacity Development Evaluation Framework https://www.fao.org/3/ca5668en/ca5668en.pdf3. A refined method for theory-based evaluation of the societal impacts of research https://pubmed.ncbi.nlm.nih.gov/32025508/