Did you know that preparation for pre-audit review requires at least six months? Truth is that even long-term projects need time to prepare for pre-audits. 

Before I delve into the essence of this blog, let me give you a brief background of who we are and what we do. Enhancing Youth Employment (EYE) is an SDC project implemented by Helvetas Swiss Intercooperation in partnership with local partner MDA (Management Development Associates). EYE Project has been active in Kosovo since 2013.  EYE’s final goal is for young women and men, including disadvantaged groups, to benefit from new or better employment opportunities sustainably. The project tries to achieve the desired change through systemic interventions in two interrelated areas: the Skills component and the Career Guidance and Job Matching component.

Monitoring and Results Measurement (MRM) plays a central role capturing the result of a project and presenting it to internal and external audiences. Attaining, classifying, and generating reliable data through the MRM system remains an essential agenda of the project. The data serves to translate our results into good stories to be shared with the public, partners, and donors, while aiming to be accountable to them all. The DCED Standard aims to ensure qualitative monitoring by improving the evaluation systems.

While the DCED Standard audits are optional, providing credible self-reported results shows the program’s seriousness about results measurement. However, due to the fact of being self-reported, the data might be interpreted as biased. Hence, the audited programs ensure that the system is accurate and is operating correctly.

As the project is in the maturity phase, the donor requested to audit the MRM system to understand if reporting aligns with DCED Standards. Our team was on edge when SDC announced they were interested in conducting the pre-audit review for the EYE Project. To be honest, the word “audit” in our culture relates more to controlling the processes rather than learning mechanisms. Nevertheless, while the pre-audit review preparation was mainly on the shoulders of the MRM Officer, this process required a lot of teamwork and involved the Intervention Managers' (IMs) active input and hard work. MRM Officer prepared the guidelines; however, the IMs provided the details and filled in the gaps. Therefore, the commitment and preparation of all team members, including management, was crucial. 

One advice is that if you are interested in undertaking a DCED Audit, remember that the preparation process will take (much) longer than anticipated.

To prepare on time, all team members agreed to stick to an action plan following a clear timeline involving divided roles and responsibilities for each. Despite technical preparation, the team needed motivation as they already had a lot on their hands - monitoring their ongoing partnerships while searching for new ones to carry-out their daily duties. Hence, to give them a good booster, we invited a sister project (PPSE- Promoting Private Sector) that passed a full DCED audit with a score of 97% on the “Must” control points. PPSE’s stories significantly impacted the teams' motivation as they shared their experience with us openly on the fundamental importance of the audit process and the learning experiences their team gained. Besides that, help from external and internal sources on understanding the key control points and where we stood by then, increased the excitement on getting the work better but also the tension on how we are going to do it.

In early April, we started with a detailed action plan prepared by the MRM Officer and management, in which precise tasks and roles were identified for each team member. Although everyone had a good understanding of most monitoring aspects and was familiar with the process - from articulating results chains to data collection - the preparation process still served as an excellent learning exercise. It was essential that everyone openly shared where they stand in terms of preparation with the management, and in which areas they needed support. Openness in sharing what worked best and why was crucial – from activity level up to the impact of the result chain. During the whole preparation process, backstopping from Helvetas regional office was of paramount importance due to ongoing support they have offered to the team. They allocated time and resources helping to increase the chances of passing the audit successfully.   It means that prior to having the real pre-audit, the team had to go through “mini audits” to ensure that the preparation is on track. While intensively working on the process, the team became more familiar with all aspects of results measurement, such as attribution - why’s and how’s, counterfactual, how to capture the systematic change, crowding in, copying, displacement, unintended effect, and most importantly to understand and reflect on the impact level – what we have achieved so far, what we aim for, and how to get to our end goal.

The preparation of technical aspects was going smoothly. However, backing up the system with evidence proved to be a challenging task. In some cases, it required going back to partners to ask for the supporting documents of the reported results - both numbers and qualitative data. This all required self-discipline, commitment, and high attention to detail. Here is an experience from the team: “Pre-audit review was a valuable learning experience for all of us. By participating in pre-audit, I have gained a deeper understanding of the importance of evidence gathering. The experience has likely given me a better appreciation for the attention to detail”. DCED Audit does not tolerate information gaps. Another member from the team stated, “The whole preparation process for the pre-auditing has given me personally a better understanding and in-depth insight regarding the MRM and its procedures”. It is of paramount importance that if information gaps exist, you need to address them promptly. This process is essential as projections for the current partners, and new ‘intervention areas’/ activities are based on proved results. During this process, the project staff recognized the purpose of the MRM system by understanding that it does not serve to only produce higher volume of data and information, but to capture reliable information on which decisions are based on. Therefore, understanding, capturing, and reporting data, helps the project think in complex environments by encouraging flexibility and continuous validation and revision of the program logic.

As we went together from early spring to September through the seven control points of DCED Standards, the team felt confident and well prepared to explain the logic of their activities through the monitoring aspect during the interviews with the consultant.

After intensive interviews, the results came in. The team felt relieved and happy after their work went through a functional monitoring system in which it met most control points based on DCED Standards.

Although the project managed to prepare for the Pre-audit by meeting the “must” criteria of the seven control points of the DCED Standards, we do not claim that the whole process went smoothly. The challenges we encountered were of different dimensions. Another team member stated: “Pre-audit was quite practical for me, which prompts you to reflect beyond the daily agenda of interventions. But it was also time consuming and challenging, especially in gathering evidence that had not been gathered in time”.  On this note, we’d like to note the following as a baseline or advice that any other project might be interested to undertake the DCED pre-audit:

First, the evidence - Each IM must back up every report with quantitative (on activity level) and qualitative evidence (on output/outcome/impact level). A report produced by internal or external sources, a TV interview of one of our beneficiaries, a story shared on their Facebook page, can provide us enough information on the impact that our project had. As such, team members should be diligent when taking part in formal and informal meetings – a quote from a primary or secondary beneficiary can back up and prove the changes happening in the system. A culture of being diligent and actively listening to our partners when they share their stories with us, can result on capturing more than the changes we thought we made! This preciseness culture of data collection should merge constantly and not only during the reporting period. Whenever you meet with a partner, remind them of the evidence (as per quantitative), and listen to them carefully to capture any unintended impact or intended results that can be attributable to the project. The purpose of the MRM system is to track the results of our actions, and to help the project make evidence-based decisions. 

Second, continue to establish a culture of transparency in the team regarding what works best and what doesn’t. The team should feel open to discuss different challenges they might face during the project implementation. There is nothing wrong with discussing and sharing when things don’t go as planned. Holding formal and informal ‘pause & reflect’ sessions during weekly or monthly meetings helps the project address gaps (if any) during the process. Understanding and identifying the gaps on time, helps adjusting the path of an activity at early stages so the project achieves the best results and higher impact. 

Third, the attribution challenge refers to links between the project intervention and the impact achieved. Respectively, to the extent to which a particular intervention attributes to the project, and why the result is attributable to the activities of a project. Sometimes, we get trapped on the activity level and forget to remember to measure the attribution on time. Whenever we are completing an activity, we have to bear in mind the end goal – why we started and how does this activity serve to the purpose of our project. As such we can ‘zoom out’ and think from a different perspective. Preferably keep and update an MRM Plan throughout the process of the result chain so nothing gets out of the loop! Since there are a variety of methods and tools to measure results, accurate planning to assess attribution is vital. Therefore, make sure you brainstorm, plan on time, but also be ready to alter your plan as needed. Don’t forget that the effect of a training held does not stop at the day the training was held or at the number of people trained– but can be traced even months later. This way you understand the ‘butterfly-effect’ of a 'not-so-complex activity.’

Lastly, the MRM system should serve the team and the management to make informed decisions and planning. As such, it should serve as a tool/ mechanism that fosters steering and continuous learning. This is easier to a user-friendly system set up with all the details of your partners, contract and milestone details, indicators, disaggregation, and all kind of information that might serve you to produce detailed and accurate reports.

February 2023
Prepared by: Vlora Kastrati-Hashani & Anda Dika