A flexible approach to impact assessment
By Prinesa Reddy, M&E Specialist
In the social investment space, measuring project outcomes and impact is often considered only at the end of a project lifespan to highlight the work that has been done. However, this loses the true value of impact measurement, which is the early identification of bottlenecks and developmental areas, and thus the opportunity to course-correct based on data. Our experience as Tshikululu has shown us the value of planning for impact measurement and data collection right from the beginning of a project to ensure that future assessments are built on a strong foundation. In reality, however, this is not always possible. We have found that we need to be flexible in how we conduct impact assessments, acknowledging any limitations while we seek to meaningfully measure impact.
Between 2023 and 2024, Tshikululu conducted an impact assessment for a global scholarship programme that provides financial support, mentorship and skills development opportunities to high-performing law students from developing countries. During the assessment we relied on the best practices we have developed over the course of 25 years. Our understanding of best practice when it comes to impact measurement is embedded in our quadrant of assessment:
• Plan for measurement: An ideal impact assessment starts before project implementation even begins. It starts with a well-designed and clearly articulated project strategy and objectives which inform which areas of the project require monitoring and measurement.
• Define the baseline: Effective impact measurement relies on accurate and relevant baseline data. This is used to conduct comparative analysis against data collected later on to assess impact.
• Standardise data collection tools and methods: Standardising what data will be collected and how allows for the most comprehensive comparative impact assessment. We recommend ensuring that data gathered during the impact assessment is comparable with baseline data, and that data collection tools enable the gathering of all relevant information.
• Allow for engagement: Factoring in direct engagement with stakeholders (in addition to text-based or indirect data collection) enhances the usefulness and quality of data collected, as it allows us to clarify any points of ambiguity, ask follow-up questions, and deepen our understanding by adding important context.
As we have previously explored in relation to fit-for-purpose M&E, however, what we think of as best practice is useful only insofar as it aligns with reality. This particular project presented an unusual challenge for Tshikululu: rather than designing and conducting a data collection process, we were tasked with working with pre-existing data. Scant baseline data was available, and the information gathered across stakeholder groups was not always comparable. We were nevertheless able to provide insight and recommendations to help strengthen the programme. We worked alongside our client to understand the context and the stakeholder groups, and drew on previous assessment reports, enriching our analysis of the available data.
Ultimately, the assessment found that the programme had contributed to improving the beneficiaries’ confidence, which has helped them access opportunities and participate more actively in the workplace. We also made recommendations about how to strengthen the mentorship component of the programme in the future, and the types of networking and support most needed by beneficiaries.
Tshikululu’s services are tailored to our clients’ diverse and unique needs. We meet them where they are to provide them with the best possible solutions – this includes adapting our approaches where necessary. While we always recommend that social investors embed impact measurement practices during project development and follow through consistently, we understand that the reality is seldom straightforward. This is especially true in a space as dynamic as social investment. Assessing impact, to whatever degree the situation allows, provides the information needed to improve performance and maximise impact. This project is an example of the value we see in navigating potential limitations to produce the best possible outcome, rather than seeking to measure impact only in perfect conditions.